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MOTION OF EDUCATIONAL TESTING SEftVJCE 
FOR LEAVE TO t^LE BRIEF AS AMICUS CURIAE 

' : * \ ' / ' ' " , 

EducationajAesting Service (hereinafter ^^ETS"l re- 
spectfully moves, pursuant to Rule *43(3) of the Rules 
of this Court, for leave to file the attached brief aviicics 
curiae in this ca^e. ^ - . 

Petitioners aiid 'the United States hive consented to 
the tiling of this bfjeff by letters dated 'November 14; 
1975 and November 25, 1975, resjjectively. Respondents,' 
by letter dated December 4, 1975, refused- to consent* 
ETS docs not urge ui)on thi^; Court* the disposition 
sought by either petitioners or respondents. Therefore, 



id 



il 



this motion and the attached brief, submitted before the 
date for completion of briefing by the parties, are timely 
presented to" the Court, See Rule 42(3K 

ETS was cha^rtered in 1947 under the Education Law 
of the State of New York as a private,, non-profit educa- 
tional corporation/ ETS was organized by three major 
educational org^izations then actively involved in test- 
ing, the American Council on Education, the College En- 

, trance Examination Board, and the Carnegie Foundation 
for the Advancement of Teaching, These organizations 
combined their existing testing functions to create a 
single organization geared to meeting the increasing 

, measurement needs in various fields of education. 

The current program of ETS refle(*ts both its initial 
charter and the substantial growth in the testing and 
measurement fields over tHe intervening decades. ET,S' 
develops and administers a, wide variety of tests for use 
in emplojTnent selection and professional certification ,and 
licensing, as well as for use in education. In the educa- 
tional field> ETS tests cover a. wide variety of academic 
'progi-ams for elementaiy, secondary/ college, and gradu- 
ate levels,' In the certification and licensing field, ETS 
has developed nearly 30 different standardized examina- 



^ The principal offices of ETS are located Jn Princeton, New 
Jerst^v^, and it maintains roRional offices in Atlaiyta, Georgia; 
Berkoloy, California; Los Anjreles, California; I^vatuston, Ulinois;* 
Austin, Texas; San Juan, Puerto Rico; and Washfngton, D C. 

- For example, ETS developed and administers the Cooperati\'e 
Testji. for elementary and secondary school students, tfie College 
Board Tests (including the Scholastic Aptitude Test) on behalf of 
the College Entrance Examination /Board , and the Giaduate Record 
Examinations/ Law Sclfooi Admissions Test,, and Admission Test 
for Graduate Study in Business, on behalf of the policy, hoards 
representing the respective graduate and professional schools, for 
prospective* graduate students. 

41- 
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tions for professional and other occupational fields.' In- 
cluded among these are two tests developed for use in 
selecting policemen/ ET& also offers advisory and in- 
structional progi*ams for educators, test users and the 
general public. 

ETS has had one limited contact with this case, in the 
proceedings below. Two current ETS employees, Dr. Al- 
bert P. Maslow and Dr. David M. Nolan, furnished affi- 
davits that were submitted and relied on by petitioners 
in the District Court. At the time the affidavits were 
'Submitted, Dr. ^ Nolan was Director of the ETS^ Wash- 
ington Office, ^nd Dr. Maslow was employed by the 
United States Civil Service Commission.^ These affidavits 
expressed professional judgments on the validity study 
submitted by petitioners, principally as to the usefulness 

Thos(» professional fields include actuarial work, , architecture, 
enprmeerinj?. law, nursing, obstetrics, ophthalmology, podiatry, 
radiology, respiratory therapy, social work and teaching. The. 
uii upatiunal fields include automobile repair, laboratory ti'chnology, 
banking, firefighting, foreign ' son ice, hospital frnancial rhanage- 
ment. merdiant marine officer's work, insurance, furniture ware- 
housing, .sffek brokerage, electrical contracting, optical work, and 
real estate sales and brokerage. 

* 0.iie was recently developed at the request of the Department 
of Personnel of the City of Philadelphia. The test measures spe.- 
eiile eogiiiti\e aljilities that were related to performance of police 
work through job analyses by both, measurement and police pro- 
fits ionals. The test has been J?oth content and concuri-ently vali- 
dated and ETS has recommended furtlrcr validation during oper- 
ational administration. A second tost was developed at the request 
of the International Association of^Chiefs of Police for use by 
local police 'departments around the coUnti^. The test measures 
speeific intellectual abilities that have been related to police work 
through job analyses by both measurement and police professionals 
and has been the subject of content validation as well as concur- 
rent \alidation in four different locales. Neither of these testes is 
similar to t-he test atMssue in this case. However, the analysis used 
by this Court to del(M-mine the "job-relAtedncss" of this test might 
affect the use of the police selection tests developed by ETS. 

, ' Dr Maslow has since joined the ETS staff as Director of the 
Center for Occupational and Professional Assessment in Princeton, 
N.J. 
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of training success as a criterion for test validation and 
whether, in light of the findings of the validity ^tudy, 
the test^t issu^ discriminates against blacks. These affi- 
davits were made by the' affiants in their individual ca- 
pacities as measurement experts; Dr. "Nolan did not sub- 
mit his affidavit as a representative of ETS. ETS^ does 
not endorse the testimony of either affiant as reflecting 
its corporate position with respect to this case. 

ETS' interest in this case arises o\\t of its extensive 
experience in developing and validating standardized 
tei?ts for employment selection and other purposes and 
its concern that the applicable legal standards be work- 
able and fair. In particular, ETS is concerned with the 
analysis employed in determining whether Test 21, used 
by the Metropolitan Police Department of the .Bistrict 
hf Columbia in selecting among applicants for entry level 
positions, is ''job-related," The .requirement of*job-re- 
latedness is a legal standard of brqad applicability to a 
variety of fact situations and, as such, it is important 
that Considerable care be used in articulating its meafi- ' 
ing and scope. ETS' interest in this legal issuers dis- 
tinct from that of either of the parties, and the attached 
amicK^'^ cinicic brief is limited to questions of law perti-^ 
nent to this is^ue that, may not be presented adequately 
by the parties. 

As reflected in the papers filed below and by their 
initial brief to this CxiUrt, petitionei^ have an expansi\Q 
view of the role Of technicJil evidence of test validation 
in deter.mining whetbi^r a particular test -js^^job-related. 
They urge that a validity study of the tesf^SSetP?^ them 
confirms both the validity of the test under profes^naJ 
measurement standards and its job-relatedness umler. 
legal standards. ^ ^ , ^ 

As indicated by their papers filed below, the respond- 
ents have a different but similarly expansive view of the 
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role *of test Validation. Respondents criticize the study 
offered petitioners in numerous' respects. Underlying 
their position is the view tRat technical validation data, 
u*ing only one of several professionally accepted meth- 
ods con>titute not only the preferred but virtually the 
only acceptable evidence of job-relatedness. 

ETS has a different view — one which, perhaps para- 
doxically, assigns a less dispositive role to statistical 
^validation evidence. ETS' experienee4n-«on^ra<*ting, ad- 
ministering,- scoring and evaluating a wide variety of 
tests related, to employment h'^censing, certification and 
.^election indicates that statistical evidence of test valid- 
ity cannot be substituted for basic analytical judgments 
about fairness and, .rationality. ETS ^-ecognize^ how- 
ever, that statistical and other evidence of 'test validity is 
important to hoth pr£>fessiorial and legaF judgments about 
the use of a particular test. ETS believes that this func- 
tion can^and should be defined with particularity. In the 
attached amic^fs curiae brief ETS presents a framework 
that utilizes technical evidence of test validity in an ap- 
propriate balance" with the analytical judgments aboivt 
fairness and i-ationality needed to implement properly 
the concept of job-relat-edness under Title VII; • 

For th(^ reasons stated^ above, ETS respectfully requests" 
t^iat this \Motion for Leave to ^ile Brief as. Amicus 
Curiae be granted. - 

Respectfully submitted. 



, ' . Howard P. Willens ^ . 

DKANNE C. SlEMEH 

Theodore S. Sims * 
Atforyieys for 
Edncati'onahTcsthifl Service 

December 19, 1975. 
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. Petitioners, 
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for the District 5of Columbia Circuit 



• BRIEF AlVTICIJS CURIAE FOR 
; EDUCATIONAL TESTING SERVICE ^ ^ ' 



QUESTIONS PRESENTED ' 

Two qufestions are pr^ented ,for review in this case. 
The first js whether a prima facie case of racial dis- 
crimination in the use of Test, 21, Has been established 
by the statistics presented by .respondents. If so, the 
second question is whether Test 21 is sufficiently job- 
related to be legally permissible in selecting among po- 
,tential employees. 
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This brief deals only with the second que^ion. It does 
not discuss the holding of the Court of Apl)eals that the 
affirmative effoits of the ^Ietropulitan Police t)epartinent 
of the District of Columbia thereinafter the "Depart- 
ment"! to recruit black officers and its success in so do- 
ing are irrelevant to respondents' showing that Test 21 
had a disparate impact. ETS recogmizes that-this Court's> 
disposition of the first question may render it unneces- 
sary to reach the istue discussed in this bvief. 

STATEMENT OF THE CASE - , 

The issue discussed in^this brief arises on the follow- 
ing fac'ts. Test 21, of which the record contains thi'e^- 
versions, App. 209-31, 232-55, 256-78,^ is a test of verbal 
ability -developed by. the United States Civil Service Com- 
mission. App. 187. It has been used by the Department 
in selecting applicants for many years and, for approxi- 
mately 20 year^, a score of 40 right ianswers out of, 80 
que.^tiuns has been the minimum acceptable grade. App. 
191. Whites achieve the requisite score on Test 21 in a 
greater proportion than blacks. App. 32-35, 

A 1967 study conducted by David L. Futi'ansky, a' 
Civil Service Commission employee, in cooperation with 

' the ^Department (the "Futransky Stu^dy'M , inquired into 
the relationship between applicant scores, on Te^t 21 and 
various aspects of ssubsequerit performance. App, 99- 
109. It used' a method referred to hy psychologists as 
"criterion-related'' validation. The critewa ^elected by 
Mr. Futrqnsky were of two 'general types: recruit ^school 
j)erformance criteria and on-the-job perfoi;mance criteria. 
The only recruit school performance xi^rion available 

^was the finrfl averag"e score"' achieved m\ ^ipht subject- 
matter examinations. There were five Dn-the-joi? perform-' 
ance criteria available: supervi>ors' ratings, commendrr- 
tions, promotions, disciplin^iy pi'O'ceedings and' separa- 

' References to the Appendix to' Petitioner.f Brit^f are desig- 
nated *'App."; those to the Appendix to the Petition for Writ of 
Certiorari are designated "CA/' 



tlon?, Mr.^ Futransky gi'ouped commendations and pro- 
motions into a single ci'iterion called i)0:^iii\e incidellt^ 
of perfonnance; ancl also grouped, low supervisory rat- 
ings, • disciplinary!' proceedings and separations into a 
single cx»itent)n called negative incideot^ 4}f j^erformance; 
He perfonned the criterion-related \aliditv. studv bv com- 
paring scores' on Test 21 to each of these criteria to^ 
determine the statistical correlation with these various 
measures. • ^ . 

■ The Futransky Study suggested, with respect to white 
recruits, the existence, of positive ' correlations betw^een 
high scores on Test 21 and subsequent ^high job perform:- 
ance ratings. App. 106-07; positive correlations between 
scores oil Test ^ 21 and positive ^ perforaance inci- 
dents, App. 109; negative correlations between scores 
on Test 2^1 and negative performance incidents^ App, 
108; and positive , correlations between performance in 
the* Depaitment's trafnihg curriculum,' as measured 'by 
;^na! averages of scores on written examinations, and 
subsequenV^ob perfonnance* ratings. App. 104-06. For 
black recruits, the Futransky Study did not disclos^e simi- 
lar correlations. .App, 104-09.. Evep for w^ite recruits, 
the Futransky Study did nof disclose, ahd it was not pos- 
sible to infer from 'the study, whethei' the' correlations 
for whites were b^sed on sufficiently^ refined analysis to 
be meaningful, App. 181, or wiiether the job perform- 
ance ratings Used as a .criterion them.^elve^ rneaningfully 
.reflected -actual job performance, App. oO!^ ;;82, 207.- 
»Both courts below proceeded on the as:>umption Uiat Te.<t 



- One obvious problem with i\iv cntefNon^of .suprr\ isoi'.s' ratmgs 
IS that it includi'd all posJti\-e ratni^:$ '( - j >throu^th ^- •} ) \^ ex- 
eluded the 0 iatin> c\ en though that rat/rt^ repre^uiited ;'ert^ti\e 
«or comp<.^tent" f)erfoi miin'ce App 104. Indii^ioir of' thi' 0 ratir>K 
witlun an index of "successfur' on-the-job peifoimancc mi^ht ma- 
terially affect the statistical. outco)T:io jf the study. ' ^ • 



21 had not been shown by the Fut^-ansky Study to pre- 
dict overall job perfomnance., CA. 17a, 49a/* ^ ^ 

The Futransky Study did establish, however, that high 
scorers on Test 21, both blacks and whites, were more 
likely^ to do \vell on written examinations administered 
during the course of the Dei:iartment's training curricu- 
lum for new recruits thereinafter V^ecruit School" k 
'At the time of the Futransky Study, Recruit School' was 
of twelve weeks' duration. App. 102^^^fi,i5;ei«;si^^-Ji^ 
syllabus of the *jecruit training cifmculum ' reflects 
that^ Re^cruit School had been expanded as of* 1971 to 
, seventeen w^eeks including two weeks of mid-curriculum 
,on-the-jQb training. App. 112. Of the- approximately 
608 hours of total instruction during Recruit School, 
approximate/ 390 hours 'or sQme^what over 2 months i 
are devotedao classrooni instruction. App. 112-13. 

The two classroom-oriented portions of the cunnculum 
are the 'Tolice (3perations Section'' and ''Laws^and Docu- 
.nients Section:" App. 114-15. gignificjgint portions of 
the Police Operations Section are devoted to understand- 
ing the 'Tolice Manual' (18 hours of classroom instruc- 
tion > and to ''Report Writing and Report Writing^Re- 
view" (42 hours of classroom instructions App, 122- 
31. Principal topics, covered in the Lawn's and Documents 
jSortion of the curriculum include the District of Co- 
lumbia Code 1 36 hours), Police Regulations ilvhoursl, 

' Traffic Regulations (23 hours), Alcoholic Beveragfe Con- 
trol Laws 111 hours), .the Rules of Evidence (20 hours) 

^ a^id the La\r Arrest; Search -and Seizure (40 hours). 
Ai)p. 13o-55". Written examinations are administered 



^^'^'The afl(ida\it.s reflect ^. some -^iisputo, which noithir court below 
re^'arded as pt»rtihent, as to .whether tho eFuti-ansky Stu(!\ e.stab- 
hshed that'Test2J did not predict job t>ei-foi-nii\;K'e ratmprs, i>osUi\e 
perfurniaiK'o incid^^nts or neprati\e perforpuinu' meuh'nts for.hlack 
.officers, App. 54-55, or whether it -ASvas. sfmpl} iticonclusne on this 
score. App. 181. , " * , ' 
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upon completion of instruction in each of these topics, 
App. 123, 131, 141, 144, 148, 155; see App. IS^, and 
each trainee is reqiiired to scare 75 percent or o^ten 
Those who fail an examination in a particular area We 
given such additional assistance, oi: instruction as msU' 
be necessary for them to achieve a^ps^ing gi'ade. Appi 
102. In this way, Recruit School failur^^>aa;;e prevented/^ 

The Futransky Study examined the correlatiolN^tween 
< scores Dn Test 21 and final averages achieved atiring 
Recruit School. It found that, for both black and white 
^ recruits. Ihose who scored higher on Test 21 were more 
likejy to graduate from Recruit School with a final aver- 
age of 85 or more.' The correlations were, from a 
•statistical "straqidpoint, significant for both blacks and 
whites. App. 19()-M. 

On these facts, respondents, intervening plaintiffs in 
the District Court, moved for partial summary .judgment. 

- In addition to an aflfidavit assessing the impact of Test 21, 
re^])ondents' Motion was based on the affidavits of two 
testing professionals to the effect that scores on Test 21 
difl not correlate with ratings ,of overall job^perform- 
ance. App. 49-57. Petitioners responded with a Cross- 

' Motion for Summa^fy Judgment, based largely on affi- 
davits of testing professionals to th^ effect that Test 21 
hcul be^n validated' as predicting relative ability to master 
the substance of the Department's Recruit School cur- 
riculum. App. 172-208; The District Court, ruled in^ 
favo]- of 'petitioners. The Court of Appeals ' reversed. 

' ' Ihv Fii^l\nnsky Study, which observed without olucidiition that 
'Tailiiio to tomplde Rrcruit School is, for all practical purposes, 
•iion-cxistcut.'* App. 100, IS tho only evidence for this conclusion. 

'App. 'l02-0;l"VVhilc on(^ of respondents' experts suj?Rcsted that 
^ the ^ijriiitirancc ef the data disclosed by the Futransky Study was 
"eont^ouiidctr' l»> the absence of evidence of whether the final aver- 
aRis \\<iT. for Heiuits who. had failed one or more of the written 
trsts. baM'd (tn the first or suhsequent taking, the record discloses 
that they were in fact based on initial taking. App. 176, 181. 
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After holding: that the re^ponS6ftl^Jlad iMde out a prhna 
fane case »of di.^crimination. in Ih&^QSe of test 21, it 
held that petitioneiv^ hall not estabn^^hecTx^^iiie^ te^t^to bfe. 
**job-i-elated," expressing ''grave doubts'' ttr^liother 
validation f£ an,^.6jn:4J^,j^nent te^ ^iigaiTist perfgrm^^nce 
in a "trainiitg program" eoulT^^^?P^§$rt*j|^*the< applica- 
ble lecral requirements. ^ ^' ' 

INTEREST OF AMi(;,us-ei?i?r?w^ 

Educational Testing Sen'ice is a private, non-profit 
educational corporation engaged primarily in the develop- 
ment, administi'ation, and scoring of > standardized tests 
in many fields. ET^ prepares tests to be us^d for em- 
l)loyment .^election purposes, including tests* to 'be used 
for selection of policemen. The specific interest of ETS 
in this ca<' is detailed in the accompanying Motion for 
Leave to File Brief As Amicfus Curiae. 

SUMMARY OF ARGUMENT 

Analysis of the requiren^ent of job-relatedness under 
Title VII of the Civil Rights Act of- 1964, as amended, 
42 U.S.C. ^ 2000e r^.s*r(/./has been con#,used by the failui^e 
to consider separately the two principal issues compre- 
hended by this requirement and by the tendency to utilize 
^^•chnical evidence of test validation alone to measure 
efntri^lianco with ha<ic legal requirements of fairne.^s and, 
i-ationality in cla's>ifications' made for employment selec- 
tion piir])Oses. * 

Courts should c*Ni^*der first the issue of whether the 
knowledge, skills oi* ofTier attributes used by an employei' 
^sify prospective employees are fair and rational 
for selection decision^. Employei's should be per- 
to meet this thi^eshokl i^equirement of job-related- 
ness b^'nl^monstrating that .the knowledge, skills or othei* 
attributes to be measured in prospective employeo? are. a 
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* fair saniple of those required in overall job performance 
or ihat ihey are key elements of job performance, the 
ab>enjce of which will create a substantial risk of non- 
?ucee^s on the job. If an employer elects to^ utilize key 
elements, jhat procedure ^should be permitted at each 
y sta,u:e of a multi-stage employment process, such as one 
^uilizing- a training program, in the same fashion as it 
is permitted in a single-stage employment process. These 
judgments ai'e largeJy independent of any test validation 
l)rocess, and should be made by assessing all the I'elevant 
e\idence^ under traditional judicial standards^fif reason- 
akl^ne-s and' fairpess. ' * ' 

" Afte>"slete mining that a measure. of job {>erfoiTn5nce 
of ^e!ected K^- elemefits may properly be used as a. iS'asis 
for emj)ir»\ment .-election, the eourtS' should then tum to 
the .separate question of the faiimess -and r-ationality of 
any .standardized, tests u>(i^. Evidence of test validity 
7TUiy be u.-eful in^ making these judgments, although there 
is no single measurement of test validity or preferred 
evidence by which the fairness a*^ rationality of the 
. xi^e of a !>articular test can be assessed under all cir- 
cunv-tance.-. The legal ^tandards for deteilnining the ap- 
propi'iatene.-s* of te>t use should avoid prescription of 
-any one method of test validation and should be stated 
so as to p^rniit the dev'elopntent of ne\?' technique^ and 
appropriate combination of , established » techniques. 

Such an, analysis of, the job-relatedness requirement' 
is fully von>i>tent \\^ith this Court's decisions and apijli- 
cable legal ^md i»rofessional standards. Both the Disti'ict 
Court -and the Coui't of Appeals in this case faired to- 
• separate the two distinct issues involved ip the determi- 
nation, of job-relatedness. BoJ:h coui'ts over-generalized 
. with re^^poct to the aiiplicable legal standards and. failed 
to consider impoilant evidentiary links in the chain of 
pi'oof o-tabli>hing job-rel!itedness in a niulti-stage em- 
\ - ; \ ^ ' , - ' . 
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plojTTient selection process. Further . elucidation of the 
jub-relatedness requirement "by this Couit is both cL\-ir- 
"~^le,,4rK]^. appropriate on the ba^is of the record in this 
case. 

ATRGUMENT 

I. THE ANALYSIS OF JOB-TRELATEDNE?;S SHOULD 
DISTINGUISH BEtWEEN THE APPROPRIATE- 
' NESS OF THE KNOWLEDGE, SKFLLe OR OTHEJl 
ATTRIBUTES USElD BY A'N EMPLOYER TO CLAS- 

• SIFY PROSPECTIVE EMPLOYEES AND THE AP- 
PROPRIATENESS OF ANY' STANDARDIZED TEST 

. • USED TO MEASURE THOSE ATTRIBUTES. 

This case pi-ovides an opportunity for this Court to 
elaborate further the requirement of **job-relatedness''. 
' established in Grirjfjs v. Duke Poicer Co., 401 U.S. 424 
(19711, and Afbcwaric Paper Co. v. Moodij, 422 U:S. 405 
Such elaboi'ation is necessan- to establish an 
analytical frannewoi'k within which specific problems 
rai>ed by individual cases can be coherently and con- 
Mstently i-'esolved. ' . 

Neither Grigf/^ nor Albemarle required a detailed in- 
qi)ir\ into the >ub^tance of the job-relatedne.ss require- 
ment/. In Gnrjfjs, the employer used two verbaPaptitude 
test.s t9 sek^ct employees for the operating divisions of a 
>team creaeratinff plant. The only evidence offered by the 
(jmi>loyer to pro\e that the tests wei-e work-i-elated \v4vs a 
vague managerial judgment that greater verbal ability 
^'generally would improve the overall quality of the work 
forcT." 401 U.S. at i3l. This Court held that Title VII 
prf^hibit.s the u.se^ of "an employment pi-actfce , which op- 
erates to exclude. Negi-oes [and which! cannot be shown 
to he I'olatod to job [>ei-formance." Id. .Since there was 
no showincT of any rela;tiohjhhip of verbal ability to job 
pQi-foimance, the f oui-t needed to examine the job-related- 
ness requirement no furthei* to conclucle that, under the 



"circumstances disclosed by the record* the use of these 
verbal aptitude tests was* improper under Title VII. 

In' Albemarle, the employer used general verbal apti^ 
tude tests in selecting applicants for employment in the 
operating divisions of a paper mill. After the (decision 
in Griggs, the employer engaged^ an industrial psycholo- 
gist who attempted to develop evidence of job-relat^dness. 
The lesultant study compared scores achieved on the 
test^^ by incumbent emplojj^ees with, supervisors' ratings 
of tho^ employees' as to^^verall on-the-job perfonnance. 
The theory was that if high verbal aptitude scores cor- 
related well ^.ith high st4>ervisory ratings, the necessary 
job-relatedness would established whether or not ajiy 

^mrticular level of verbal aptitude was actually i^equired 
for the job. This Court found the validation study to be 
defective in a number of basic respects and concluded 
that, since the minima^ relationship. sought to be proved 
could not be supported by the study, and no^ other evi- 
dence was offered^ the requirement of job-relatedness had 
not been met. On the facts in Albemarle, the Court was 
not required to examine the nature of the relationship 

. nought to be proved or to consider evidence other than 
the test validatipn study. 

As increasingly complex cases in the employment test- 
ing^ jfteld coniQ before'J:he courts, it becomes highly desir- 
able for this 'Court to 'articulate further the standards 
'.'for determining whether' the basic ;*job-relatedness*' re- 
quirement has been met. Properly construed, the deter- 
mination whether a particular test challenged under" 
title -VII is -**job-related'\ requires answers to two, basic 
que>tions: 1 1 ) Has the employer selected an attribute or 
skill to be measured? which is reasonably related to likely 
success Cpr lack of success) on the job? and <2)*If so^ 
is the te^tt at issue properly designed and* used to meas- 
ured the particular attribute selected by the employer? 
Examination of these^two questions will assi-st jn sharp- 
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ening the relevant issues and in defining ai}i)roijritite 
roles for the test validation expert and the court in mak- 
ing the required ''job-relatedness'' determination, » Such 
an analysis is not only consistent with Griggs and Albe^ 
marie but is also supported by the rele.vant professional 
guidelines set out .by the American Psychological Asso- 
ciation \ and the relevant legal guidelines set out by the 
Equal Employment Opportunity Commission.^ 

A. Employers Should Be Permitted to Meet the 
Threshold Requirement of Job-Relatedn'ess by 
^ Demonstrating a Legitimate Interest in One or 
* More Key Areas of Knowledge, Skill' or Other 
Attributes^ 

The requirenient of job-relatedness as first articulated 
by the courts, understandably described job performance 
in general terms. Many employers assess their em'ploy- 
^ees with respect to their overall competence in perform- 
ing the tasks requii^ed by the job, frequently by refer- 
ence to the observations of supervisors. A number of 
ld\V(^r counts, in describing the job-relatedness require-, 
ment under Title VII, have used the phrase ''joverall job 
performance" as 'the standard^ to which any, selectid^ 
technique must be related.' Although it is plain that a 



•'^American Psychological Association. Standakds fok Educa- 
tional AM) PsYorOLOGlCAL TESTS 11974) (hereinafter the *'APA 
Standards"). The APA Standards arc. a revision of the Stand- 
ards FOR EnirCATIONAL AND PSYCHOLGGICAL TESTS AND ^MANUALS 
(19G6), referred to at 29 C.F.R. § 1607.5(a). 

" E({ual Employment Opportunity Comnnysion.^ "-(Tuidt^hnes on 
Ennployee Sek^ction Procedures," 29 C.F.R. § 1607 vt S(^. (1971) 
(hereinafter the "EEOC" Guidelines?'). . 

Sec I'nited Statts V. X^ity of Chicago, 1585 F. Siipr^, 54:5, 555-56 
iN.D. III. 1074); Vulcan Soc'y of N, Y. Fire Dept. w'f.^.C , ^60 
F. Supp. 1265. 1272*74 ( ^.h.N.Y . ) , ^aff'd in nli vavt pari, 41)0 F.2d 
\m (2d Cir. 197:n. C/. Smith v. City of K. Cleveland, M?. V. Supp. 
il31. 1148 (N^D. Ohio 107in, revd oth other grounds sub 7iom. 
Smith V. Troyan, 520 F.2d 492 (Gth Cir. 1975). Many case^i did not 
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measure of overall performance may provide an accept- 
able basis for establishing the necessary jbb-relatedness, 
any suggestion that such a measure is indispensable to 
establish job-relatedness should be rejected by this Court. 
The legitimate interests of employers and the existing 
legal precedents demonstrate that alternaiive means, re- 
lying on key , elements of job performance rather than 
on overall assessments of success, are equally permissible 
and desirable under Title VII, ■ 

1. There is a legitimate int^erest in focusing on pne 
or more key elem^ts of job performance. 

An emptoyer's principal legitimate purpose in classify- 
ing persons at any point during the process of employ- 
ment selection is to maximize the chances of selecting 
those persons who are Hk-ely to be successful at their jobs 
and to minimize the risk of selecting those who are' not. ' 
In many employment situations, ^attempfmg to maximize 
the .chances of selecting successful: employees involves a 
different, and more^ difficult, assessment than minimizing 
the risks of selecting those who are likely to be unsuccess- 
ful. Because of the difficulties in defining overall suc- 
cess, employers frequently find it useful to identify one 
or more k^ey elements of job proficiency the absence of 



ivach this question simply because there was Little or no e\idence 
of job-rehitedness at all. E.g., Bridgeport Guardians. Inc. v. Islvm- 
hers of Brulgeport C.S.(\, 354 F. Supp. 778, 79O-0;5 (D. Conn.), 
uff'd in nlrrayit part, 482 F.2d VMV.^ (2d Civ. 197:5) ; Chance v. Bd. 
of Examiners, :m F. Supp., 20:5, 216-10 (S.D.N.Y. VMD.aff'd 
<\rxH F2d 1167. 1 174 (2d C^ir. 1972) ; Kirkhmd v. N.V State D.C.S., 
:574'F. Supp i:5Gl, i:572"76 (S.D.N.Y. 1974), aff'd 520 F.2d 420 (2d 
Cir. 1975). 

/'A classification can be fair if it pro\ ides? some reasonable level 
of eertuint> that, as the standard ib? applied to a Rroup of prospec- 
ts candidatos, the decisions made taken as a \\\\o\v will M'rve 
lejritimate employer objectives. No classification cim ensure that 
every individual decision using the criterion will be correct in 
terms of the legitimated purpose. 
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w^hich increases the risk that a prospective employee will 
not. perform successfully. 

Successful overall job performance is frequently difTi- 
cult to define. Various combinations of individual 
strengths can overcome various combinations pf individ- ^ 
,ual weaknesses. Thus,, it is difficult to agrje^ on some 
"meaningful measure of overall job success e\4n' f or iobs 
requiring. fairly sitjipld skills: / . • ' 

''For, example, the success pf production workers 
might be gauged in terms, of the number, of units of 
work turned- out per-^y, the proportion' of com- 
pleted units that pass inspection, wastage,- an^d main- 
tenance- ofc tools an^ equipment Clerks rhight be 

*^ rated by their Superiors/ in. terms, of quanti^ty ,of 
work, accuracy, ^and initiative- . , . [Iln most biases 

' no single criterion can be takep to give"^ a complete 
descxnption of job success because each of- them 
measures some important /nd pertinent j^hase of per- 
formance/' ^ . 

Even for relatively simple jobs, the variety of ways in . 
which proficiency is defioied may depend on different and,. 

more importa'Vitly, unrelated characteristics: ' 

> ' 

'*The fast typist may not necessarily be -the most 
accurate, and the bus operator who has th^ fewest 
accidents may not be the one who best maintains 
his schedule. So questions arise as to whether such, 
unrelated kinds of performance ,can be meaningfully 
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E. GiiiSEMJ, Tnf: Validity Of Occupational Ai'titudk 
Tksts 22-23 (19G6) (hereinafter cited as "GiiiSKLLt"). St'c aUo 
L C^RONBACH, Essentials of Psychological Testing 4ia-lG (:Trd 
Oil 1071) (hereinafter cited as 'X'RONBAcn); A. Anastasi, Psy-* 
("iiOLOGicAL tLSTiN'G 417 (3rd ed. 19G8) (hereinafter cited as 
''Anastasi") ; R. Thorndike & E. IIagkn, Mkasuhkment and 
P^VALi'ATioN IN PSYCHOLOGY AND EDUCATION 234 (1955) (herein- 
after cited as "Thorndike & Hagen'*). 
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combined into a single over-all index of job suc- 
cess/' ^ 

It is even clearer for" more complex jobs that: 

"Success is never unidimensional. Particularly in 
Jiigh-level positions, there are many patterns of suc- 
cess. Teachers, for example, may. excel in different 
ways: one develop^ into a friend^ and counselor for 
youth, one' stimulates independent thinking in the 
jfew bnghtest students, one overcomes the blockings 
that cause failure among weak students. To try to 
score these^ types of perfonnance on a single scale 
is pointless — one loses infonnation. . , ^- ^ ^ 

Similar problems inhere in the implementation of meas- 
ures of overall job success even when they are available. 
• Foi* example, supervisory ratings used to assess overall 



GuisELLi at 2:^ See also Croxracu at 443-44. 

Cronhach at 44 !^ Src Bridgeport Guardians, Inc. v. Members 
of HndKi^port (\S.C., ;^54 F. Supp.'778 (D. Conn.), aff'd in, relevant 
'pn}l, 482 P^2d i:^:'.:^ (2d Cir. 1973), where, in considerinR the job- 
reliitedne.^s of a test used to select pqlice recryits, .the Court ob- 
served that: 

"[WJith jobs of <%nore complexity, especially those requiring 
exerei.se of judgment, there is not likely to be agreement on 
\vhieh enteria are truly indicative of successful job perform-' 
anee, nor has ihv art of evaluation accjuired such' precision that* 
mi^asurements of job performance gan always be readily or 
accurately made." !^54 F, Supp. at 789. 

See (tho Castro v. Heecher, V. Supp. 930 (D. Mass. 1071), 
aff:d ni part and rev'd in part, 459V.2d 725 (1st Cir. 1972) : 

/'One might hazard a guess that reliable aptitude tests cannot 
be (it signed for candidates for thi* Presidency, C'ongress, the 
Supreme Court and many lesser public and private offices. The 
' * . I'easons include . . the greater significance of integrity, 
sticngth of charactlj^'r, altrachve personality, emotional sta- 
bilit\, wisdom, judgfnent, sympathy, social imagination, intui- 
tion, common sense >^nd good fortune/* [VM F. Supp. at 942- 
4;i, n.9. 
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performance on many less complex jobs may frequently 
be biased and suspect/ - - ^ 

The risks of non-success, however, are easier to define 
conceptually and -ultimately easier to measure. Almost 
every job invoh'^s one or jnore key elem^ts— either par- 
ticulai; knowledge, skills, abilities, personal' characteris- 
tics, or other attributes — which, if absent in a prospective 
employee* ^incre^ise substantially the risks of non-success. 
Examj)les are readily' avaiiable : the prospective biology 
teacher who has an effective classroom manner, relates 
well to parents and keeps records efficiently **but who 
lacks basic knowledg6 of biologj^; the prospective engi- 
neer who has the requisite knowledge of mathematics 
and engineering i)rinciples, is skilled in the handling of 
measiirement and drafting tools, has th*e verbal ability 
.to commtinicate with others, but who cannot read basic 
blue[)rints; the prospective bank teller who can do the 
computations required, operate the '6ffice machinery and 
deal with the public but whose personality is sucb^ that 
gambling is an irresistibly attractive activity. In each 
of these .cases* and in many others, the assessment of 
candidates in terms of an important 'dimension of work 
l)erfonnanco can as^sist'an employer, in identifying those 
for whom the risk of non-success is relatively high.'* 

' , 1 

"TfiouNPiKi: & IIacf:^ at 2^\\ CnosMCU at 571-74: Bn'to v. Zia 
(a. 478 F.2d 1200. 120G-07 (lOth Cir. 1071^) ; Vulcan Soo'y of iV.V. 
Firo D(^pt. \, (\S.C., 400 F.2d 387, 395 & nJO (2d Cir. 107:^). 

"Srr ('astro v. Beochor, :VM F Supp. 9:^0, 943 & n.O (D. IVIass. 
1971 ) , (iff (I m rdrvaiit part, 450 R2d 725 ( 1st Cir. 1072 ) ( "It would 
not ho astonishinjr to loarn that the most that intrlliKinco or apti- 
tude tests (an be counted upon to accomplish is to winnow out ihi\ 
oh\i()Usly in(f>mpct^it poli<^o candidates") ; Pennsylvania \\ O'NeiU^ 
:M8 F. Sypp. 1084, 1001 ( E.D. Pa. 1972),,a/f'rf bj} m rq}^ divided 
ravrt. 47:^> F.2d 1020 C^rd V\r. 107:0 i en banc) ( "fpTestin^cxpertJs 
can reach judp^ments as to whether .some applicants may be so dbM'- 

.1 

28 . ■ 
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Tests, may as a practical matter be more useful - in 
assessing^ applicants in terms . of particular characteris- 
tics for ^ which an employer may have a legitinfiate con- 
cern. Tests can be related in a more meaningful way to 
particular work-related proficiencies'' or aptituSeS*'"' than 
to overall job success. Ultimately it may be possible to 
combine a series of tests for particular job-related char- 
acteristics inlo batteries that may provide a more com-- 
prehensive appraisal of individual applicants.'' But the 
development of an overall battery, for even a single job 
or class of jobs, may be an expensive and protracted 
process.'"* which even then has practical limitations whiqfi 
not all employers can afford- to bean^^ 

Consequently, there is a practical need, particularly 
*for employers that make hiring decisions frequently, to 
be able to appraise potential candidates in terms of 



ously unqualified on the hi^is of their performance on the examina- 
tion that th(Mr acceptance on the police force would not assist in 
the validation of a passing, grade and/or would constitute an un- 
acceptable risk.") . ^ 

* ^'"Sf'f', r.fir., An'ASTASI at 424-34. 

'''See, e.g., CronbaCH at 417-21. . ' 

^' See AnaStasi at 413-16j Cronbacii at 410-17. See also 
Bucknerv. Goodyear Tire and^Rubbcr Co., 339 P. Supp. 1108 (N.D 
Ala. 1912), afd, 47(}, F.2d 1287 (5th Cir. 1973). 

'""E.g., Cronbach at 414*21„4'43-44. 

'"As one authority observed, in order to develop an appropriate 
test battery the U^. Air Force; 

"went to the trouble of sending through training a 1300-man 

* random sample of all eligible I'ecruits, even though th^jy knew 
in advance that the majority would fail. .\ . 

"The Air Force took costly risks in sending an unselected 
group into training. Neither the employer nor the public want.^ 
to ::?ee unscreened men in responsible positions. The employer 
tied by .seniority rules may properly refuse to hiuo low scorers 
that he will be unabJe^to get rid of. All the costs, obvious and 
hidden. muBt be weighed in deciding on the proper scale and 
duration of a tryout/' Cronbach at 412. 
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emploj^meTit-related characteristics more narrowly de- 
fiited than overall job success. A classification of poten- 
tial employees using rationally selected and ^efensiBle 
key element^ of job performance- serves the Same legiti- 
mate purpose as a^classifi cation based on measures re- 
lated to^overall job performance and both methods should 
be equally acceptable under Title VIL 

2. Use of key areas of knotvledge, skill or other ^ 
attributes as criteria for classifying employees ' 
. '/s, consistent with Existing case precedent, ai(d 
guidelines. ' • 

The foXlndation of this approach w^s recognized by 
this Court in Albemarle Rapey Co, v. ^^oqdy, 422 U.S. 
405, *433 (19751, when It faulted ^the employer's assesi^- 
ments of employee perforh>ance becatis^ there ' was no 
evidence -that they wer^ sufficiently related **to the Com- 
pany's 'legitimate interest in job-specific ability . , . /' 
i Emphasis added.) 

The EEOC Gui(Jelines also recognize the propriety of 
classifying potential employees on the basis of: 

"important elements of worK behavior which" tomr* 
prise or 'are relevant to the 'job oi;'jobs for which 
cl\ndidates are being evaluated.'' 

* "Elsewhere, the Guidelines explicitly provi"de that: - 

''The work behfiviors or other criteria of employee 
a/:lequacy which the test is intended to ^predict or 
identify . . may include measures other than actual 
work proficiency,' such as training tirne, ^supervisory 



-^>29 C.F.K § lG07.^t(c). This provision of the EEOC Guidelines 
*Was (Tidorsed by this Court m Albr77inrlr. 422 UiS. at 431. 
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rating.-, regularity of attendance and tenure." 29 
CF.R. S 1607.5(bM3). 

^ < 

It is reasonably tleav that the Guidelines do not 'purport 
-to 'exhaust the range of permissible work-related, beha- 
viors for which an employer may legitimately bl con- 
cerned. Instead, they provide generally that: 

''Whatever criteria are used they must represent 
major or critical work behavioi'S as revealed by care- 
ful' job analyses/' 29 CF.R, § i607.5(bi (3). 

Moreover, the Guidelines do not condition test use on the 
development of^ batteries designed to provide ^ compre- 
hensKe candidate evaluation. They provide only that' a 
test must be a valid measure of > 'at Jeast one relevant 
criterion'/ of work-related importance.''. 

The professional" g-uidelines-mal>e a. similar provision 
for selecting key elements: , ^ • ' - 

*'Te^t users might define the performance domain 
/ . f in ternis of appropi;j[ately detailed and compre- 
hensive-job analyses..'. . . The* perfonnaDce do- 
main, wbyld need definition in terms of .'the -objectives 
of measurement, restricted perhaps oilfe^to critical, 
most freqtient, or prerequisite-. w6rk' behaviors/' 
* APA Standards at 29; . > ' 

.This' approach tQ the use of standardized tests 'is re- 
flected in United: Slates v. Norih Carolina, 400 F. Supp. 
343 ( E.I].N..C. ' 1975 ) i three- judge court'), wiiere the 
couH considered a' challenge to the use of tests in. certify- 
ing'prosi)octive teachers. In that case the state had not- 



_ 2\) (• F,R § 1G07.5(^^) ( r). the Guidelines caution. hcA.vever, that 
,when> an cmplo^Vcr usrs *'a single test as the sole selection dcMce*^' 
use of the test "will hv senitinizvd cloj^ely wherf tfiat tost is \alid 
apainst on!} one component of job i)erformance."-29 C.FK S 1007 5- 
(c)(1). , . • ' ' / 
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performed any job Analysis for the job of teaching' and 
' had not attempted any assessment of candidates for eer; 
tification that related to overall job perforpancer-^e 
court held that: 

/'[I]t is beyond argument that' -the State of North 
Carolina has the right to [single out the key element 
of knowledge and] adopt academic requirements and 
written achievement tests designed and validated to 
disclose the minimum amount of knowledge neces- 
sai-y to effective teaching.'' - 

The court distinguished between the emplo^Tnent selec- 
tion and certificati^qn processes, pointing out tha^ an em- 
ployer ma;y select only tlie best from among qualified can- 
' didates while -a certifying' authority must certify all 
**those who aVe minimally competent, 400 F. Supp. at 350. 
However, the court pointed out Ihat^the requirement of a 
^ job-related criterion by which to make the classification ^ 
is tlie same in both cases. 400 F. Supp. at 351 ^n.8. 

.The threshold question whether an employer has iden- 
tified a Sufficiently important work-related "attnbute is 
largely independent of technical questions of test valida- ^ 
tion. Indeed, the propriety of selecting employees on (he 
basis ^of 'any particular attribute does not depend oi> 
whether fhat attribute is appraised by a test. 



— 400 F. Supp. at 348; 'citrng other case? i^n which particular 
knowledge had be^ singled out as the criterion on which a classi-, 
frcation was based: Dent \\ West Virginia, ,120 U.S. 114 (1889)* 
.(mcdfcuie) : Step})ons v. Dennis, 293. F. Supp. 580 (N.D. Ala. 1068) 
(pharmacy); Lomlwirdi v. Tauro, 47^) F.2d 708 (l.st Cir. 1072).' 
cert, denisd, 412 U.^. 910 (1073) (law); Graves'v. 'iMinno.sota, 272 
U.S. 425 (1926) {dentistry) ; Williamson v. Lee Optical Co., 
lis. 48;{ (:1055) (optometry); England v. Louisiana Bd. of 
Medical Examiners, 246 F. Supp. -09;^ (E.D. La. 1065), ajf'd mem., 
^84 U.S. 8^5 (1966) (chiropractic medicine) ; and Miluer M Burson,. 
320 F. Supp. 706 (N.D. (3aj.i970,) (driver training instruction). 
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This Court's reference to '*job-jspecific ability'' in Albe- 
marle confirms that it is not possible to fraipe a compre- 

, hensiv€ legal definition of what particular attributes are 
of legitimate concern for particular jobs. It is, moreover, 
reatonably clear that both the quantum and type of evi- 
dence ^necessai-y to identify the attribute will vaiy from 
situation to situation. Few would quarrel with the propo- 
sition that secretaries, should know-how to %pe or bank 
tellers how to add and subtract.^' 'Equally few Would 
dispute that social worker^ should have a basic knowledge 
of socioIog}^ that nurses should have some essential 
understanding of relevant aspects of medical practice, or 
that air traffic controllers and airline pilots sho\ild be 
familiar .with the^ pertinent FAA regulations. It may 
npt be difficult for a prospective employer to demonstrate 
the propriety of such concerns by direct testimony of 
managers or supervisors familiar with the jobs in ques- 

. tion. 

^' In some contexts, however, establishing the' legitimacy 
or an employer's concern for some work-related attri- 
butes may requiVe more extensive evidence and more care- 
ful judicial review. This might be .the case particularly 
with less« concrete characteristics such as ''thoroughness," 
''initiative'' or "accuracy" * and with less p/oficiency- 
onented aspects of work behavior such as r^^gularity of 
attendance and punctuality. In dealing with less Con- 
Qrete charactenstics, demonstrating the relevance *of the 
employer's concern might require Use of careful job analy- 
ses carried out by professional* industrial psychologists. 
See 29 C.F.R. § 1607.5(b) (3). 

Expre>s recognition of the necessity for. case-by-case 
deteimination of this issue would be a useful elaboration 
of thi;s Court's ,(}ecisiohs in Gnggs and Albemarle, By 
.articulating the standards to be used in appraising an 



-""Sec APA Standards at 29, 61. 
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employer's selection and use of key attributes in classify- 
ing employees, this Court would be recognizing the prac- 
tical needs of the range of employers who are subject 
to the requirements of Title VII. 

3. The use of key elements related to job perform- 
ance as classification criteria is permissible at 
each stage of multi-stage employment selection 
processes, * 

Employers often must make employment selection de- 
ci^jions in stages, gi'adually separating out those who are 
unlikely to succeed. This may be required by the nature 
of the job, the relatively large investment to be made in 
new employees, or other considerations of employment 
efficiency. The legal standards with respect to fairness 
in employment selection should not work to the relative 
disadvantage of an employer who uses a multi-stage se- 
lection process. Such an employer should be pemitted 
to demonstrate that each classificatiqn made is rationally 
related to his legitimate purpose in maximizing the chance 
of selecting successful employees or minimizing the risk 
of selecting unsuccessful employees. The use of one or 
more key areas of knowledge, skill, ability or other at- 
tributes should be subject to the same legal standards 
irrespective of the particular selection process used. 

Multi-stage selection processes typically involve one or 
more training programs. Because of the specialized na- 
ture of the job, the employer may not be able to recruit 
p.qtential employees who already have the precise attri- 
butes needed on the job.'* If the absence of one or more 



2* A training program need not address every key clement in a 
job. Itcjyi sene its legitimate purpose far the omplo>er if it tramps 
pro.spective employees in only one pr several key elements of the 
job. There may be more effectivo iays to address the j-emaining 
key elements j^iyyt covered by the tiainiiig program; sueh as com- 
pletion of an academic program or An-the-job. experience. 
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of these attributes is substantially related to the risk of 
nun-^ucce^^ on rhe job, then the employer hsu? a legitimate 
interest in constructing a training program to produce 
the desired attribute, and in selecting only those trainees 
who are likely to be able to acquire the desired attribute 
during the training program. , ^ 

Since a training program may be expensive ^to design 
and operate, an employer has a legitimate interest in ob- 
taining the most effective use of the resources, he has 
de\oted to it.- The process of learning in trainirig may 
require skills different than work on the job *uniess the 
job itself requires continual learning of new '^material. 
For thi.- » reason, the employer should be permitited to 
apply the key eletxient analysis to the training piogram 
itself and to select skills or abilities without whicli there 
is a substantial risk of non-success in the trainii;(g pro- 
gram. If the training program is itself fairly related to 
job performance, then the key elements needed (o mini- 
mize the risk of non-success inihe training progi'am will 
necessarily be logically related to job performance. 

Ha\ing narrowed his focus in this fashion, the^mp^oy^i^ 
should be permitted to set up a qualifying c(iterion or 
several qualifying criteria based on the k^y elements 
that relate to the risk of non-success in trahiing.'" This 
approach was recognized in Grigg.s when tho Court found 

-"'An important ancillary aspect of^this analysis is that the 
ni« lliiid usrd to impart knowledge or skills in a/t raining program 
l>e leasniiable in li^ht of^the content of the trj^ining prggrani. In 
t>n far as possible, the method should be no/ more acaciomically 
oriented or abstract than is reqinred by th(> job-related subject 
mnlti'r. - i 

~ If, m addition, training was protracted, 4n employer shoukl be 
I)erniitted to a.sM'rt that the ability to co^^plete traiilinjr'" within 
ih(i specified time was itself a work- related concern. 29 CF.R. - 
§ 1007.5(1;) ( n. ' ^ ; 

-'•Xa (ine (lualifyinK criterion need cciver all key elements and 
t\pieajyS' tlifreient eriteiia such acadei;i^ie course work, prior work 
expeiii iKv. .standai dized tests* oi^ demonstfatjoh of particular woik • 
skills Will be used to measure the various key 'elements. 



the employer had failed to sustain its burden of proof 
because neither test **was directed or intended to measure 
the ability to learn to perform a particular job or category 
of }obs:\ 401 U,S. at 428 (emphasis added K Qualifying ' 
criteria may properly be addressed to the ability to leai-n, 
if that learning process is reasonably necess^yy to job 
perfoiTnance, ^' 

At each stage of such a multi-sta^, selection process, 
the employer must be able to demon^'ate that the* en- 
teria utilized are related to legitimate employment objec- 
tives, either directly by reference to successfuUjob per-^ 
formance or indirectly by reference to another stage in 
the selection process which itself beai-s the hecefesary I'ela- 
tionship to job performance. However, if this require- 
ment is satisfied at each stage of the process, there is no 
further overarching requirement with respect to the 
whole process. 

The case law developed in the lower courts with re- 
spect to multi-stage selection processes is not inconsistent 
with this analysis. 

Such an apiJi-oach to evaluating training success as a 
work-related attribute^ is set out in BucJnwr V, Goodyear 
Tire and Rubber' Co., 339 F, Supp, 1108 (N.D, Ala, 
1972), aff'd, 476 F,2d 1287 (5th Cir, 1973), In Biickner 
, the court considered the job relatedness of a battery of 
employment tests used to select applicants for an indus- 
trial employer*s fOui'-year apprentice training program* for 
skiUed craft jobs. The court accepted evidence of the 
test batteo^'s validity based on comparisons with training 
success. The court obseiTed that accepting sUch evidence 
meant that: 

'*[T|he relationvship between the apprenticeship pro- 
gram and the filling of craft job vacancies tjien be- 
comes also a matter for inquiry. , . 339 P. Supp. 
at 1114 n.4. 

Thus, in a subsequent portion of its opinion the court" 
examined the .content of the training program to ascpr- 
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tain whether it was ''job-related/' It found a number> 
of the academic courses included in the training program 
to be insufficiently related to the job and^irected their 
deletion from the training curriculum, l^also directed 
removal from the battery of ^ entrance tests those that 
had been validated only against success in the deleted 
courses. 339 F. Supp,^ at 1124, In doing so, the court cau- 
tioned, however, that it. did not mean to 'imply that a 
training program is necessarily '*^.ubject to intensive vali- 
dation studies such a& outlined in 29 C.F.R, g 1607," 339 
F. Supp, at 1124 n,19, - ' ^ 

^ ether training, program decisions to date involved a 
lack of evidence sufficient to determine whether the traiji- 
.ing program was intended to impart essential work-re- 
lated knowledge or skills or that it was adequately de- 
signed to do so,-' For example, in Pems-ylvania v. O'Heill^ 
348 F. Supp. 1084 {E,D: t?a, 1972)., afd m relevanj^pari 
by an equalhj divided court, 473 F,2d 1029 (Sd Cir. 1973) 
{cn banoAhe court considered evidence of jpb-relatedness 
based, on 'a claim tlfat an entrance test correlated with 
butcess in a police training program. In rejecting such 
evidence of job-relatedness, the court w^as- influenced, by . 
an absence of the ''shoxying of any correlation between 
success lit the Police Academy and effective performance 
on the job/' 348 F. Supp. at 1090. But it appears that 
the .employei' made no attempt to establish by competent 
e\i(lence that knowledge of the relevant laws was essen- 
tial for effective job performance. Similarly, there was no 
« — * \ - ' _ ^ 

There are, of coiiTsfs decisions which rejected trainjng pro- 
Kram \ahdalioii, not on the ground that tniining success was an 
impermissible employer concern, but on the ground that the eni- 
pl^ji'r Had not shown a tost to predict training success. Sec, e.g., 
Pji idgepori Guardians, Inc. v. IVIcmbers of Brid|:eport C.S.C.,- 354 
F. Supp. 778, 71)1- (D. Conn.), aff'd in relevant paU, 4g2 F.2d 1333 
(2d Cir?. 11)7:5); Oincers for Justice v. C.S.C, ^11^ Supp. 1328, 
l:;37 ( N.I). Cal. H)73 ). Sec alao DozTor v. Chupka, 395 F. Supp. 836, 
815-lG, 852-;^:? (S.D. Ohio- 1,075), in which one of two tests was 
.simply nul \ alidated .and the second was impropei'Iy combined with 
the results of a physical agility test for .purposes of validation. 




Widence that the training program curriculum was ade- 
quately designed to impart the essential knowledge or 
that passing grades .on the training tests constituted a 
fair measure of mastery of the draining curriculum. Jn- 
de^d, the court pointed out that: 

''^TJhe correlation made by [the employer's experts] 
may be misleading. There is no evidence as to' the 
y ,type of examination administered at the' Academy. If 
in i'act tests of the type contained on the entrance 
^ exafiiination were used, a high correlafeion could be 
anticipated even if an individual had not mastered 
the training material." 348 F. Supp. at 1091* 

*The decision in Harper V. Mayor and City Council of 
Bdlkiinor^^ 359 F. Supp. 1187 (-D. Md.), modified and 
aff'd sub nam. Harper V. Klosters, 486 F.2d 1134 (4th 
Cir. 1973), involved an attempt to correlate scores on an 
entry test .with success on a training program for fire-^ 
man, recruits. In Harper the court was confronted with 
the possibii^ity that the test|, u^ed to assess mastery of 
the trainingr curriculum were identical with the entrance 
test. 359 }^\' Supp. at 1202-03. However, there was, 
as' in Pcmuiijlvania v. O'Neill, no evidence that the re- 
cruit schooljwas adequately designed to impart skills 
and knowledge which the fire department might legiti- 
mat(?ly consider to be work-related. Moreover, in re- 
jecting the^ evidence of training school correlation, the 
court was influenced by the haphazard nature of the test 
itself.'' Based upon its 'Wn perusal of the quality of' 

The court observed that pivpaiuitmh of ii number of tlie entry 
W^iH had been "the solitary work of a Service] Cummiasion 

persoiuiel teehnicfan''" who **Kicked professional training in test con- 
^struction." Ho whs, nnorcover : ^ ^ 

^*as a Commilssion employee, unfamiliar with Fire Department 
work. Workiibg principally on his own. ho constructed a test 
that wa.s u.4ed as an alternate to [another ) exam from 
iOG2 to ll){>*). . . . In 1971, the Commission employf^e was again 
givenithc job of constrnctnfg a Fire D^)artment entry level 
test. This tinie he had the advice of a study panel which had 
been appointed, by the /ire board in an* effort to lessen tho 
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the exaiVis in question, the court concluded ''t^at if the 
tests were valid, it would only be by sheerest chance/^ 
359 F. Supp. at 1187: Thus; the court never reached 
the question of the legitimacy of training success on the 
basis of a 'record demonstrating that the training cur- 
riculum was fairly designed to impart work-related 
knowledge. 

One decision, Vniiei States v. City of' Chicago, 385 F. 
Supp. 543, 556 (N.D. Ill 1^74), has suggested that em- 
])loyiiiGnt selection, tests must be related to actual job 
performance to the exclusion of any measure of training 
projri'am success. The case involved an* attempt to use 
ti'aininj; success for police recruits as an independent 
w;ork-rehUed consideration. But^ as in Pennsylvania v. 
CyNcill, there was no evidence tQ establish either 
that the training curriculum was designed to impact 
knowledge essential to effective police performance oi' 
that acquisition of the relevant knowledge was fairly 
reflected in success on Police Academy exams.'^' Thus, 
the court's observation that overall job ^erfo^-mance- was 
the only legitimate employer concern was made in the 
context of a record tl;at gave *the court ^tio opportunity 
to considei' the in^lependent relevance of trajning success. 



racial impact of the entrance tests. Ttiough he took their advice 
o!i many points it was not expert advice^ and the actual choice 
of questions remained hu-gely his own/* :^59 F. Supp. at 1197. 

The fact tKat the training examinations were thems^elvcs not in 
the record was siiggestod by the court's observation that ''one would 
cxpi'ct a correlation between performance on written ^exams/* ;?85 ^ 
F. Supp 55G. without considerinpr, as m the Hanger case, that the 
academy examinations might be .essentially diflertnit from the em- * 
ployment selection tests. ^ 
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B. Employers Should Be Permitted 't6 Meet the Re- 
maining Requirement of Job-Relatedness by Dem- 
onstratinj? thut the Key Areas of Jolj-JPerformance 
Selected as Criteria Are Measured by Valid an'd 
Appropriate Standardized Tests. 

)^ 

The first step in aniflyzing the fairness of an estab- 
lished selecti'on process is to evaluate the legitimacy of 
the employer's concern for the knowledge, skill, ability 
or other attributes selected as a standard, of Hkely suc- 
cess or non-success on the job. Once the key element has 
been identified and determined to be fairly related to job 
performance, there -must be an evaluation of the means 
used to assess prospective employees in terfns o^ the key 
element. The^fairness of the measurement process is 
distinct from the legitimacy of the e^ployei-s concern 
fo]' the key element to be measured, and should be sepa- 
rately considered^'*^ 

Standardized tests are only one measurement tool that 
employers can fand do) use. If particular knowledge is 
important, academic credentials are often used as a 
measure^ of its presence or absence. If prior work ex- 
perience is central to an employer's objectives, refevencos 
from former employers are often used as a measurement 
device. If personal attributes are important, interviews 
with the prospective employee are a likely means of as- 
sessment. The legal standards applicable to the measure- 
ment of k^y elements should be fair and workable for 
''both test and non-test measurement devices. ^ 

^1 Plainly, a fair criterion can be unfairly appliqd. Thii^ would 
occur in a situation whero possession of certain knowiedpi^ is a 
fair criterion, btit only':^0 percent of the (luostions on the tost iisod 
to measure the criterion are devoted to this specified kno\vIedf3:(\ 
Equally so, an unfair criterion can be fairly applied, as in a case 
whore a police department (establishes kiiowiedpe of Renaissance 
art as an entrance reQuiroment and t^e test adheres scrupulously 
to the specified subject. 
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When an employer has selected a standardized test as 
the measurenrenXtool, courts have looked first to the 
technical evidenceX^f test validation presented by the 
employer in determining the fairness of the measurement 
process. As the law has developed in the lower courts, 
there has been a tendency to rely almost, exclusively on 
suclf evidence and to prefer evidence of criterion-related 
in making, a judgment as to the appropriateness 
)articular test use. Although evidence of test vali- 
has an important ,role, ETS believes that there 
^e-Tnfeasiimnent of test validity by which* the 
^ fess and rationality (rf- a particular test use'caij^be 
assessed in all circumstances. Judgments based on rea- 
son and all the relevant evidence are as appropriate and 
necessary where tests are used a<3 a basis for .the clas- 
sification of individuals for employment purpos^ asjn 
Itny other type of classification case which comesi before 
this Court. • • ^ \ ' f ^ 

^ ^ 1. Evidence of- validation is properly used to deter* 

mijie the measurement capability of a test. ^ 

There gi^several professionally accept^ techniques 
for •ascertaining 'hvhat [a] te§t measures and how well 
it does so'' and c6nfirming that "the test actually meas- 
ures w^hat it purports to /neasure." These techniques 
are what measurement professionals refer to as. valida- 
tion. Of the methods reflected in the professional litera- 
ture, three— 'content/' "criterion-related/', and ''con- 
struct'' validation — have the widest degree of professional 
sanction. All three "^re specifically recognized by both 
the EEOC Guidelines, 29 C.F.R. 5 1607.5(a), and the^ 
APA Standards fat 25-31). T\v,o— ^^cQntgnt" and "crir 



'^AnaWSI at 09 (omphasis in original); CronbaCII at 121-22. 

"Anastasi at 28; CroNbacit at* 121-22: Thoundikk & Hagkn 
at 108. ^ . > . 

- J. • . 
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terion-relatecV' validation — have|the greatest practical 
relevance to the validatign of tesis for' employment, selec- 
tion"" purposes given the current plate of \aIidation tech- 
nology.^ • ' ! ' ^ 

Content validation is most comlrionly used Vith respect 
to proficiency or achievement tests /'designed to measure 
how well an individual' has jnasiere^ a specific skill or 
.course of stucty/' ' This method of vaHdation is an expert 
asse^^sment of the design of the testfto ensure that it ade- 
quately samples, for exami)]e, the topics of the x^ourse of 
study. Proper content validation requires the test devel-, 
oper to define the body of knowledge 'or catalogue {he 
range of skills '(commonly rrferred to as the ''perform- 
ance domainal to be assessed and to select the test ques- 
tions so that, taken as a whole,' they constitute a cross- 

" "Construct" validity focuses "on a broader, more enduring, and 
more abstract kind of behavioral description" than cithor "con- 
tent" or "critorion-rehited" validity. It is an' attimpt to ascertain 
"tKe extent to which [a] test may be said 'to mea.sLii'c a theoretical 
construct on trait." Axastasi at 114-15; Croxhacu at 122-25. Thus*; 
unlike "content" \alidity, which attempts? to tonstiuict a ti'st as a 
n'presciitative sample of some defined concrete area, or "criterion- 
related" \alidity which is the examination- of 'corn' 1 at ions between 
test scores aod specific external Vhaviors, "construct validity" is 
more concerned with ultimate traits such as "intelliRence. mechan- 
ical comprehension, verbal fluenci^ . niuroticisni. and anxiety" ro- 
tfected in- scores on particular tests. AxASTAi^i at 111. It iiuohi's 
elements of both other forms of \alidation arfd holds peihaps the 
greatest interest for theoretic&l investigations into standaidizcd 
tests. ^ ^ 

Users of tests for oinployment selection purposes gonerally ha\e 
more particularized needs and require more specific judprments w- 
^rardin^r the measurement capability of a tost than are a\ailabl(' 
from construct validity under current measurement techruciucs. 
iro\ve\*er. construct validity is an effort to combin(^ leasoned 
judgments about fairru^ss and rationality with statistieal indices 
of coi-relation in a reliable framework for analysis and is ultimately 
likely to be more useful than^ either content or critiiion-related 
validity standing -alone. 

■^"^ AnaStaSI at 100; CuoNnACir at 124-25; sec also TIIOUNDIKE & 
Hagen at 109-10. 

/ 
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section or ''representative sample'' in both substance and 
difficulty of the defirjgd perfonnance domain.^' For.exam- 
pl^, the validity of a test batter}^ prepared to assess t^e 
outcomes of the training of medical school graduates 
would depend on the extent tc^ which the battery sampled 
a cro^s-.-ection of what, in the pooled judgment of "experts, 
medical school curricula ought to impart.'"- In essence, 
content validity requii-es systematic examination^ both of 
'*the test and the methods used in its prepal-atibn/' 

Criterion-related validation is air inquiry, undertaken 
after a test has been fonnulated, as to whether it is possi- 
ble ''to infer from a test ^core an individualVmost prob- 
able standing on some other variabl^^alled a criterion,'' 
and is most often used to validate aptitude tests. It is an 
attempt to ascertain the extent to which test performance 
correlates with performance in some external endeavor.^' 
The process of • criterion-related validation consists of 
gathering data on both test performance and criterion 
.performance and, by a process of statistical analysis, as- 



^" A\AST\si at 100-02, CR0.\B\CHat 122-23; Thorndike & Hagen 
at 110-12. APA Sta.ndaKds at 28. 

• Thor.vdike Sl Hagkx at 111. 5ec APA Standards at 45-46. 

'M.. Cromucii, Essentials of Psychological Testing 364 
1 2d wl. IDGO). 5rc aho Anastasi at 100; APA Standards at 29. 

APA^ Standards at 2G; Cronbach' at 122, '126-27; Anastasi at 
105; (7/.<,v> TiioR,\i)iKE & Hagen at 115-16. 

Two Ki'^nruniy retoKnized subcategories of cnterian-related vali- 
dation an; predietiNe Und concurrent validation. "Predictive \arul- 
ity" jTi'm>rally 'in\olve9 a comparison of the performance of the 
subjeclrt of the study, fir.st\)n the test and later on the cnterion. 
"Cuncwrrent \ahdity" involves the correlation of th'e performance 
of tfie swbjpcts ()f the study on the test and, more or less simul- 
tanciHisly. on the criterion. Anastasi at 105-06; CroNBACII at 122; 
TlIOKNDIKE & l^AGEN at 115-16. 
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certaining .whether the correlation between the two may 
be regarded as ^'significant/' 

2. There is nq single' method of evaluating t,€st 
validity or preferred evidence bg which the fair- 
neas and rationality of the use of a test can be 
assessed under all circumstances. 

The measurement profession s concerns for test validity 
are of almost equally long standing as the widespread 
use 6f standardized tests. The .current APA Standards- 
have their roots in formal technical recommendations 
promulgated by that organization and others more than 
20 y^ars age' Textbooks and professional literature 
of a quarter-century and more ago reflect extensive in- 
quiries into test 'validation.*- 

The purpose of the validation work done over the 
years is to provide information that will assist test de- 
velopers to construct more useful tests ajid to assist test 
users in interpreting test scores, two principal measure- 
ment concerns. This v-alida^ion work has not: generally 
been concemecT wfth developing standards that permit a 
Jecision as to whether a test rtay be used at all,; the mo^t 

"Surnilicance" is used in -the statistical st^nse, that i>. the likeli- 
hood that the eurrelatiun could ha\^^ occuried by chance. Set n.49 
infra. 

' American Psychological Association*. Technical FfkcoM- 

MKNDATIONS FOR PSUIIOLOGICAL TtSTS AND DIAGNOSTIC TU IINIQILS 
( 11)54) . AMERICAN EorCATlONAL RKSKAIiCH ASSOCIATION & NATIONAL 
CorxCIL ON EDl.CATIONAL~^^*ASe£t:^tENT, TECHNICAL RECOMMENDA- 
TIONS FOR ACHIEVEMENT TESTS (195517"^^ . 

^- Sr/\\.r;./ R. Thokndike, Personnel Sutction (1040 i : A. 
AnaSTASI, PsycHOL0(;ical Testin^P. 120-51 i2d ed. 1054) ; F Frll- 

m\n\. JlIEORY AND PUACTICE OK PSYrilOLOGICAL TESTING, 2G-4 1 

' ( 1955 -TllORNPlKE & IIagen at 108.-23, 256-GO. 
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relevant legal concern/^ To some extent, the interpreta- 
tke^as>istance found in evidence of test validity assists 
in malang:. the legal jadgment. Such evidence may be 
nece^^ary to the legal judgment; it should nrrt be substi- 
tuted for the legal judgment because it was not^ developed ' 
or intended for that purpose. * ^ • 

\et a number of lov^er courts have been .developing 
what appear? to be an unrealistic reliance on e\idence 
of te^t validity. .Most of the lower c6urt testing cases 
rhus^iar have involved the use of aptitude tests. The 
APA\ STANDARDS reflect a "preference, for measurement 
purpo>e>. for criterion-related validation of aptitude 
tests. The EEOC Guidelines have adopted this prefer- 
-ence for criterion-related \alidity as a legal standard'for - 
oil types of tests. Section 1607.5faj protitles that em- 
ployers may establish a test to be ''job related" through 
the u>e of: 
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' A> thf .-VPA Stwdakds recognize: 
' AlmcNi an\ test can be useful for some function and in some 
-il.iation-. hut e.en the htst test can ha\G damaging- conso- 
(|utnci.'> if u.>od inappropriatc'Iy. . . . 

"This diKumunt-is prepared as a technical guide for those with- 
in the .sponsoring profession; it \s 7wt^\ntUin as law. What is 
intended is a set of standards to he used in part for self- 
e\ahiation by te.^t de\elopers and test users. An e\aluation ot/ 
their competence Voes not rest on the literal satisfaction of 
e\riy rtl(*\ant pn>\ision of this document. The individual 
.standards are statementvS of ideals or goab. some ha\ing 
. "prif)rit\ over otliers. Instead, an evaluation of competence, dv- 
puids on the degree to which the intent of this document 
has \Hvn satisfied by the test de\Cdoper or user." APA Staxd- 
\Hr)S at 7-8. 

'^APA St\ndards at 27-28. The distinction between aptitude ^ 
te^ts tind aehU'\ement tests is largely based on- the u.se of a test 
rather than its intrinsic nature. A test of knowledge of chemi.'Ttry, 
for example, might normally be used as an achievement test to meas- 
'ure hfKV miKb vhl'ml^tl^v> th(- candidate had learned. Howe\er, the 
same te.sr of chemistiy migl^t be used as an aptitude -test to select - 
candidates for admission to ad\aneed placement or graduate study 
programs. 
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**Evidence of content or construct validity, as defined 
in' [the APA STArjDARlis] , . . . where criterion- 
related validity is not feasible/' '\ ; ' 
This leap, from a preference for criterion-related validit> 
for measatement purposes with respect to some tests to " 
a legal standard with respect to all tests has teen re- 
flected in some lower court decisions/" Such^cases have 
usually involved aptitude tests and an evident lack oi 
fairness and rationality such that the same 'result would 
probably have been reached under any analysis. Other 
courts, however, have been m9re sensitive ^to the need 
for flexibility in considering validation evidence/" Any 

*'T'he absence of full public scrutiny of and comment on the 
Guideliijus before promulgation may ha\e enjfendered some am- 
biguitv. One such ambiguity invokes the use in § 1607. 5{aj of. the 
term '"feasible." The term "technically feasible." on the other hand. 
IS used at a number of points in the Guidelines (e.p.. §§ 1607.4(a) ; 
1607.5(b) < 1) ), and is defined with great rigor in § 1607.4fb). It is 
not altogether clear whether § 1607.5(a), in using the term "feas- 
ible.** meant "technically feasible," as some courts ha\e assumed 
to be the case. E.g., Douglas v. Hampton, 512 F.2d 976, 986 (D.C. 
Cir. lOfo); Harper v. Mayor and City Council of Baltimore. 359 
F. Supp. 1187, 1202 (D. Md.j. modified and tff'd sub nom. Harper 
V. Klosters, 486 F 2d im (4th Cir. 1973). The indication in section. 
^1607.5fa) itself that "evidence of content validity alone may be 
acceptable'* suggests that some less rigorous connotation was in- 
tended bynhe use of "feasible" r^her than "technically feasible:*' 

^^Eg Kirkland v. N.Y. State D.CS.. Wo/R2d 420. 426 (2d 
Cir. 1075) ; Rogers v. Infl Paper Co.. 510 1340. 1340 fSth Cir. 
1975 J - Chance v. Bd. of Examiners, 330 F. Supp. 203. 21,G i^.D, 
N.Y I9?l). aff'd. 458 F.2d 1167. ^177 & n.16 (2d Cir: 4f72) ; 
HaiT)er \. Mayor and City Council of Baltimore. 359 F. Supp. 1187., 
1201 (D, Md.). modified and aff'd sub no'm. Harper \. Klosters. 486 
K.2d 11:^4 Uth Cir. ^1973); Fowler v. Schwarzvvaldc^r, 351 ¥. Supp. 
t21 (,D. Minn. 1972): Pennsylvania v. O'Neill, 348. F. Supp. 1084. 
1090-91 (.E.n. Pa. 1972), aff'd by an equally divided court, 473 
F.2d 1029 (3rd Cir. 1973) (en banc). 

E.g., Vulcan SocV of N.V. Fire Dept. v. C.S.C.. 400 F.2d 387. 
.^94-95 (2d (^ir. \ Bridgeport Guardians. Inc v. Memluu's of 

Kridgeport C.S.c! 354 F. Supp. 778. 791 (D.'Conn.), aff'din rde- 
'vant part. 482 F.2d 1333, 1337-38; Kirkland v. New Yt)ffc^.^te 
D.C.S.. 374 F. Supp. 11^61 (S.D.N. V. 1973). fcv'd in part on oUici 
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standard which appears to rely exclusively on statistical 
evidence of criterion-related validity will be both illogical 
and unfair to employers who have used st^ndardizecj 
tests in a rational and professionally accepted manner. 

Criterion-related validity, has inherent limitations as 
an exclusive basis for legal judgments at^out test fair- 
ness. To eptabMsh criterion-related validity it i^, neces- 
sary only to establish a moderately positive correlation 
between two sets of data for a given group of persons. 
One set of data is the score on a test; the other set of 
data is the ''score" on some other performance criterion. 
The coi'relation range runs from —1 (perfect positive 
correlation where high scores on the t^st always coincide 
with high scores on the performance rating) to,-^l (per- 
fect negative correlation where high scores oji the test 
always coincide' with low scores on the performance 
rating ).*^ If the correlation is 0,30* 0/ above and the 
sample size is at least 45, then the measurenient profes- 
sional will conclude that criterion-related validity is es- 
tablished at a level that is statistically significant/' 



fjrmmds, 520 F 2d 420 (2d Cir. 1975) ; Officers for Justice v. C.S.C., 
V :>,71 F. Supp. 1328. 1336 ( N.D. Cal. 1973). Cf. Smith v. Citv of* 

. (Mevoland. 363 F Supp. 1131. .1148-49 (N.D. Ohio 1973), revd on 
^ a(}ur grounds ^ub nam. Smith 'v. Troyan, 520 F.2d 492 (6th Cir. 
1975): Coopersmith v. Roudebush, 517 F.2d 818, 824 (D.C. Cir. 
1975). 

A corn lation trn fficiont ranp:inj? within these limits is jarenerally 
used in psychometric research. Other coefficients of correlation 
are occasionally used that have a different range. 

^ Stati?ti(:al significance is ordinarily determined from math- 

ematical UihM and is stated in terms of a particular -^e^l of con- 
fidence*' A statement that a correlation is siprnificant atMhe .05 
level drnotcs that chances are no grcater than five out of a hundred 
"that the true correlation in the population is 0 — in other woros, the 
probability of such a correlation occurring by chance is no greater 
than five timys in a hundred. R. Fischp:r, STATISTICAL METHODS FOR 
Ri:.sK\RCii WOKKI'KS 209 / 1 1th cd. 1950). Most psychometric re- 
search employs a 05 level of confidence. Occasionally a lower .10 
level or a higher .01 level may be used. 
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But a closer look at such statistical evidence indicates 
its inherent limitations. First, for a correlation of 0.30 
only of the reason for the variation in scores oi^the 
performance criterion is attributable to whatever the 
standardized test measures. The remaining 91 is ex- 
plained by some other factor. The 0.30 correlation does 
.not pro\ide any information about whether the remain- 
ing 91^^ is e>^plained by factors more relevant' than the 
factor measured by the standardized test or how vital- 
the factor measured by the^tandardized test is for 'ade- 
quate job performance. Seecmd, the 0.30 correlation does 
jiot help to make a judgment about what the standard- 
ized test actually measures or what the perforrhance 
criterion actually measures. The standardized test may 
be titled '''Verbal Aptitude" but it may in fact measure 
nly spelling capability. The performance criterion may 
be labelled /'supervisor's rating o^ overall job per-f^rn> 
ance" but it may in fact measure only whether or not 
the supervisor likes the person rated. Therefore, the 
0.30 correlation may be e\idence only of the fact that 
good s])elling correlates favorably with supervisors' per- 
sonal preferences. "Di^gedangers of depending 'exclu- 
sively on correlational >^^ditJ^>i|l. evaluating a test have 
long been recognized py meaVire^lent experts.'"" 

Plainlj/^'for^se legal* -sb/ndard of fairness and 

rationaliiy, ci;ileinon-r^^ should be supple- 

mented K^^^id^ce to^^upport at least two ancillao^ 
propositions; flW-wh^t .;nTes^ndardized test actually 
measured; and 1 2 r^hat Hheyperfomance criterion ac- 
tually measured. The aeUurf^ibject matter testeil by the 
te^^t might be proved through a content validity study or, 
in the case of a simple test, through inspection.'' 

^'^E.f/., riulliksoii^ Infriyisio Vallditij, 5 Amkkicax Ps^niOLOGiST 
„ 511,514 (Octol)or. l9/)0j, • - ■ " 

'^Compare, for oxami^le, decisions in which a courts judKmont 
invalidating use of a test ha^ been jnftuQnced-by its own inspcc- 
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E\idence with respect to the performance cntex-ion might 
comf^ from industrial psych ologists or other measurement 
experts or from those experienced with the employment 
^etting'from which the performance data was collected. 
Once these fundamental requirements have been satisfied, 
there still room for infomed judgment as to whether 
the particthlar correlation reflected in the cnterion-related 
validity study is appropriate in light of the employer's 
objective in using the test. ' 

The use of criterion-related validity as ''preferred evi- 
dence" uL^o could lead t<y results that do not satisfy legal 
standards. For example, tlje^^^test" might consist of a 
^erie^ of multiple choice 'questions ^bout family back- 
ground and income level. The coil-^ation between white 
middle cla^s economic statiis and success on any igiven 



tioVi of the tt^'St Harper v. May^fJir and Citj Council of Baltimore, 
.150 F Surp. 1187, 120^> <DmUi, viodififd and aff'd svb wmi. 
Hai*p«^'r \ Klovtu-^, 48G ¥M ll?>A Uth Cir 1973): Ca.'^tro v.. 
Hcf^chcr. 'r,4 F Supp iv,\0. 912 ij). Mass. 1971 i.CaJ?'</ in rdivant 
pnft\\y^ F2cl 72f? list Oir, 1072), Bridgeport G^^rdiaiiS, Inc. v. 
'M. nij^t'r- r)f irrid?rnoi-t CSC, ^.54 F Siipp 778, 7^91 (D C'onn ), 
(UTdn^ rdftnnt pnrl,'^^^2 F.2d (2d Cir. 1973). 

- In makiii^r thi< determination, th^ naUiro of 'the pcrfoiinance 
critmon particularly important. ^ If thf particular paptitudc, skill 
(»r altility w of ^^reat innponancc, and the risk of non-success i.s high 
UY the rr^iijlts of non-success extrenneli; costly < either in .social or 
raonetary trrms;. then use of a^ test with a rdati\c4y low coriTla- 
tion might he lationally related to the legitimate purpose of mini- 
mizing that i-Lsk On the other hand, where the risk is relatncly 
in tt'rms of occu'rrence or co.st, then the necessary rational re- 
lationship to an employer's legitimate purpo.M- might require a 
higher coi relation. * 

It IS also important to consider tho siz.e of the applicant pool in . 
rclalinn to'the numhrr of candidates to be selected. If the size of 
tlu' pool IS large and the numher of persons to he selected is small, 
then \i'iy Ifiw correlations ma\ he p^-actically useful because the^^ 
proMdc greater efTicieiicy than would a random selection of appli- 
cants. 
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empIoATTient performance criterion might be quite high/'^ 
But it would seem unfair to pemit an employer to hire 
only white middle class persons on the. basis of that 
statistical evidence of criterion-related validity. More 
subtly, a test may be labeled as an achievement or ajTti- 
tude test with a specific subject matter content, but in 
fact be constructed in a way that measures highly eso- 
teric material with which only a very narrow segment 
. of the population would be familiar. Such a test might 
show statfstical evi9ence of very high criterion-relatecl 
validity but, again, should not be acceptable under a 
legal standard? ' 

With these technical limitations in niind, ETS believes 
that the courts must look beyond the confines of measure- 
ment technology^ to make both of the 'fundamental judg- 
mentv^^ involved In the job-relatedness'detei-mination un- 
der Title Yll, This* \s cleBrly so with respect to ' the \ 
empl0\;er's use pf a key element of job perfoi^mance as'^a 
'selection criterion; <^vidence of cjjterion-related validity 
may assist in making this judg^i^nt,. but cannot substi- 
tute for it. It may alsa be peraiissible, and under some 
circumstances it is obligatoiT, to look beyon(J the confines 
of measurement technology to Yriate tjae second critical 
judgment w^hether the test fairly measures the key eJe-^ 
V ment of job performance. To do otherwise would result in 

an abdication of judicial responsibility and an excessive 
reliance on measurement technology which, most profes- 
sionals in the field would concede is not justified. 
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'In one much-cited study, generally positive anc^^ significant 
correlations were found between vanou.s measures of cranial ca- 
pacity and Rrade.s received by freshman coUeKO students during 
their first .semester. C. 1Iall,'Aptitude Testing 134<^G>1028 ). 
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3. The legal standards for the measurement aspects 
of job-relatedness should permit development of 
new techniques and appropriate combination of 
recognized techniques. 

Properly used, standardized tests provide an objective 
measure that is. open to inspection and easy to compre- 
hend. From a standpoint of fairness, they are usually 
preferable to subjective measures because of their sub- 
sUintially higher reliability 'and potential for higher 
. validity. ' Therefore the legal standards applied to st^nd- 
aj-dized tests should not be so much more stringent than 
those applied to subjective assessments that the use of. 
tests is made economically or otherwise impractical. 

Beyond thes.e concerns is the need ttf state legal stand- 
ards in a way that does not impede progress within the 
/ measurcmaent profession or prevent employers from using 
new validatloii techniques as they aVe^ developed. Valida- 
tion techniquejr have been revised and refined over the 
years. Various th'eoi:ies have obtained currency in the*^ 
profession and then declined in acceptance as more per- 
sua.-lve theories were develo}3ed. The me^snrement pro- 
fe.-sion ha> coritinued to search. for more accurate w^ays 
to define what tests measure ahd describe how they 
measure it. "Experience teaches tliat the preferred 
method of today rnay be the rejected one of tomorrow." 
yvlcan Sociciy of N.K Fire Dejmrtmmtv, C.S.C., 490 
F.2d ^304 (2d Cir. 1973). 

New validati.on methods are currently being introduced 
that use th^^^judgments of large juries of persons who 
are experienced on , the job about the propriety of 
the content of a test and the 4iklly level of perform- 

Wilson. A Si,r(»nd Ij^ok at G)i(jc/s v. Duke Poinr Company: 
llinnninf >nni^ on Jnh Tti^tnuj, Discuminnflrm, and thv Role of the 
F^hnd ComfH, 58 V\. L. Rkw 811.87:^ (1972) ; Note, Empiomnent 
Tvfitlng' th( Aftermath of Grujfjs' v. Duhc Power Company, 72 
COLUM. L. lihv. 000, 924 (1972). 
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ance on the test of a miniinally qualified person,^ This 
method has evolved from theoretical' discussions in the ^ 
technical literature and has been made possible by the • 
availability of computers to manage and assess the data. 
It has substantial advantaps over .current techniques 
because th^ use of a large number of judgments elimi- 
nates' the effect of personal bias that may be present even 
in experts, and the use of ^persons who are experieaeed 
on the job or in training persons for the job provides 
an additional touchstone of reality, ^ 

Such new developments would be adversely affected if 
the legal standards for jobrrelatedness are stated exclu- 
sively in terms of current validation techniques. A 
souncfer appreach would be for this Court to define the 
Jegal standards more generally in terms of the traditional 
requirement of rationality and fairness that the neces- , 
saiy 'flexibility and room for experimentation can be 
preseiwed. 

II PROPER ANALYSIS 0^ THE JOB-RELATED- 
NESS ISSUE IN THIS CASE WILL FACILITATE 
ITS RESOLUTION AND PROVIDE NEEDED 
GUIDANCE FOR THE LpWER COURTS, 

The Dacis case is the first employment testing case to' 
come before this Court which requires careful analysis of 
the requiretnent of **joB-relatedness'' in the context of a 

'••W. Crawford, Assessing Performance when the Stakes Are 
HiKh.iMarch, 1970 4 Paper presented at the meeting of the American 
EdueiitiQnal Research Association, Minneapolis) ; J. Fremer, Crite- 
non-I^oferenced InterpteUitions of Survey Achievement Tests, 1972 
(Kdueational Testing Sernce, Test Development Memorandum 1^- 
1 ) , N. Luebkc. A Practical Method of Doterxnining a Criteilon 
Score for Criterion-Referenced Measurement, February, 197;U Paper 
presented at the meeting of the American Educational Rese<'M'eh As- 
soication, New Orleans) ; IC Sparks, IMemorandum on the ('onfer- 
ence on the United States Department of State Foreign Ser\ko 
Examinations' in French, Spanish, German, and Russian (JuFy 12-, 
14 1957); Nodelsky. Abs(4ntc Grading Stayidards /r)r Objective 
Tests] 14 Educational and Psychological Measuulment 319 
(1954). r.O . 
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multi-j>tage employee selection process. Ndther court be- 
low engaged in the necessary analysis; neither properly 
apjlraised the test validation evidence in the case; and 
neither resisted the temptation to offer- excessively broad 
generalizations regarding the application of this Court's 
prior opinions to the case at hand. In many respects, full 

^ explication of the ''job-relatedness" issues raised by the 
Dans cabC is handicapped by the limited record below 
and the sumnjaiy posture of the case as it comes before 
this Court, Subject to this caveat, the Davis case pro- 
vides an opportunity for this Court to establish the 

'neces>ary parameters of the '*job-relatedness" standard 
under Title VIL 

A. Is Know ledge of Laws and Other Law Enforcement 
Subjects i\n Appropriate Key Attribute of Police 
Work which Can Be Used by the Department in 
;^ Selecting Police Recruits? 

Although neither court below addressed this central 
qUL\>tion in precisely these terms, their opinions appear 
to approach the issue from different perspectives and, not 
surprisingly, suggest ^different cojiclusions. Two basic 
questions are presented by Davis. First,. does the knowl- 
edge identified by the Department bear the necessary 
relationship to successful police performance on the job? 
Second is the" Dcj^-tment's recruit training- program 
reasonably designed to impart, that knowledge? The Dis- 
trict Court appears to have answered both questions in 
tlK.^ affirmative; the ,Coui:t of Appeals appears to have 
condudt»(i that neither question need be addressed in the 
absence; of convincing evidence validating either Test 21 
or the recruit training program in st overall perform- 
ance on the job by policemen. Depending upon this 
Coui-tV \iow of the .adequacy of the evidentiary record, 
this Court coul'd rii]e mSavor of either petitioners or 

53 ^ 
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respondents or remand the case for further proceedings 
below. '^^ 

L Knowledge as an appropriate key attribute of 
police work. 

In the absence of any correlation between performance 
on Test 21 and overall job pft'f onnance, "J the Dp.vis\:ase 
requires consideration of the propriety of identifying a 
key attribute— knowledge of certain law enforcement sub- 
ject^as a basis for selecting policemen. 

Neither the Department's pleadings nor the proof below 
defined the knowledge considered by the Department to 
be a key. dement of effective police work. Nowhere is the 
risk assessed of £he consequences to the public if police- 
men lack such knowledge. However, it may be i^asonable 
to infer that^the Department, based on its collective ex- 
perience, has concluded that effective iX)lice"work requires 

Because this case involves a governmental employer, the analy- 
sis used to determine whether the necessao' rational relationships 
ha\e been established will be the^ same regardless of whieh party 
has the burden of going fonvard. If this Court determines that ad- 
verse racial impact exists based on the sUitistical evidence ofTered 
by the respondents, then under Title VII the petitioners have the 
burden of going fonvard to establish that the test is being used 
properly. If the Court determines that no adverse racial impact 
exists, then under the Fifth and Fourteenth Amendments the re- 
.spondenU have t[ie burden of going forward to esUiblish that the 
tost IS being used improperly and that, accordingly, the classifica- 
tion based on its^ise is irrational. 

There^ippcars to be np dispute that the Fntransky Study dis- 
closed no significant correlations between performance on Test 21 
and o\erall job performWe. App. 181, para. 8. IIowc\er, it is ilp- 
propnate to note that this conclusion does not support the proposi- 
tion that a signilicant correlation between performance on Test 21 
and o\erall job performance cannot be established. Futraiisky se- 
lected as a criterion of positi\e performance hupervisors' ratings of 
I I through 14 He did not include ratings of 0 although that 
rating means "eflectivi' or competent" penformjmce. App. 105. If 
the 0 ratings had been included in the criterioii/Zigainst which Test 
21 scores were compared, a positne correlation agaiiist job per- 
formance for all officers may have been established. Sec App. 181. 
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knowledge of (1) the laws and Jtxegulation^to be en- 
forced, and (2) certain rules, principles, and techniques 
with respect to law enforcement. It appears equal^ rea- 
sonable to infer thatu^e, risk of failure on thie J<yb in 
the absence of such knowledge — at least in terms of mak- 
ing improper an-ests and failing to make proper arrests 
—would be substantial v ^ 

The District Court emphasized the ''responsibilities and 
expertjs^ required of , modern police officers in a large 
metropolitan city . . anB, after summarizing the 
syllabus of the recruit training course, concluded that the 
daily significance of police skills "demanding reasoning, 
and verbal and literacy skills is borne iDut in the crucible 
of the criminal trial court/' 348 F.Supp. at 17. Charac- 
terizing law enforcement as "a highly skilled professional 
service,** the District Court obviously -concluded that the 
knowledge taught in the, training curriculum was suffi- 
ciently important and related to effective, police wx^rk so as 
to peiTnit its use by the , Department ia sd^cting among 
recruit candidates. .On a very limited e\identiaiy record, 
in sh(}\% the f)istrict Court reasoned that it was not 
necessary to correlate perfonnance on Test"'21 with over- 
all job perfonnance' in order to meet the requirement of 
"job-rclatedness*' under Title VII, and that the necessary 
relationship was evident 

The majority of the Court of Appeals disagreed. It 
emphasized that there" were no coiTclations, at least for 
black office!^, 'between reci-uit school performance and 
job-proficiency ratings. '512 F.2d at 963/ The Court of 
Ai)poals, therefoi'e^ never considered thd relationship em- 
phasized by the District Q^urt between the knowledge 
taught in the 'training pragrani and the knowledge re- 
quired for effective police work. Its failure, to do so 
appears to reflect its view^ that ''job-relatedness'' can^ 
demonstrated only on^the basis of correlation with overall 
job pei'formance,' but it. is. possible that the Court ^of 
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Appeals merely concluded that the record in the Dam 
case could not support such a finding. ' 

Consistent with our analysis of the job-relatedness re-, 
quirement, ETS submits that it would be entirely appro- 
priate for the Department to identify knowledge of laws 
and enforcement rules as an important dimension of police 
performance on the basis of which its -employees can be 
properly selected. The presence of such knowledge may 
not ensure successful police performance, which like most 
complex jobs requires a mix of personal and professional 
skills, but its absence probably will involve a substantial 
^ risk of po^ police performance which the Department 
can properly seek to avoid. This is a deteriniiiation as to 
which technical issues of test validation are largely ir- 
relevant. " The principal question for this Court in con- 
•sidei-ing this issue is w^hether hie present record provides 
^ a sufficient basis for such a determination to 'be made.'" 



''^ Rt'jrardloss /)f the Court's underlying: rationale for not consid- 
Vrinpr thjs question, its failure to do so led It to oxtensiM' further 
discussion of "trainability" and "training proKi'ams"' which raise ^ 
separate and^ t^(iually important issues reK^rdinp the Vjob-relatid- 
ness*'* determination, S( c discussion ivfru at pp. 45-47.. 

'-•''Sec (\astro v. Beecher, 459 F*2d 725. 735 (1st Cir. 1972). in ' 
♦ which' the Court concluded that the requirement of a hiph ?ichool 
education was. established to be job-related by the fnulin^'s of tha . 
Report of the President's Commission on Law Enforcement arid 
'Admuustration of .Justice. Tin: riiALLKNCK of Ckimi. in a Frke 
SociKTY U'itinp: pp, lOG-10 of the Report). 

ETS takes no position as to whether the Department has; on 
this record, adequately pro\e(! that certain knowledge is an ap- 
propriate key element of successful police work. 

The Department has cited cases and othei* authorities for the 
propositioa that a certain le\<el of knowltnlpre important foi* suc- 
cessful pohce work. Petitioners' Brief at 18^2. The rvidentiai-y 
prol)lem is whethei*. if certain key elements of knouledpre for ^suc- 
cessful police work could hv identified, the Department can rely on 
tlio (existence of thus possibility whet.her or not. at the time the 
traininjj: program wa.s established, the Department actually had 
identified such key lilements of knowledge. Although nut discussed 

; . ob 
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2. Adequacy/ and reasonfibleness of the recruit 
training program. 

If the 'Court concludes that knowledge of particular' 
subjects" can properly be emphasized in the selecfiofir- 
process by the Department, it is necessary to examine 
further whether the particular training program used by 
the Department i? fairly and reasonably designed to im- 
part the nece^a-ry knowledge/^'^ The District Court, based^ 
upon its review of the training program syllabus, coa-^ 
eluded that the program did teach subjects that were 
reasonably related to the?' Department's legitimate inter- 
est in policemen with the desired knowledge and exper- 
tise. 348 F. Supp. at 17, The Court -of Appeals did not 
directly addi-ess the question. It did, however, indirectly 
cast doubt upon the Departrnent's training program in 
the course of its discussion of the propriety of correlating 
any test with ''trainability" or a **training prograpi." 
In so doing, the Court of Appeals may have departed un- 
necessaiiily from the applicable precedents and the specific 
questions raised in this rase. 

The Court of Appeals seemed to suggest, in the ab- 
sence of any correlation against overall job performance. 



in these tcrms^ the Pepartment is plainly relying? on the existence 
of a Recruit School syllabys containing subjects logically related 
to police work as the basis for an inference that at the time the 
Recruit School syjlabiis was devised, someone at the Department 
identified certain k^cy ejementa^of knowledge believed to be essential 
to sncceiiiiful -police work. ^ 

One test'of reasonableness involves the design of the training 
program with i-espect to the nature of the subject matter knowledge 
to be imparted An employer who has identified knowledge of cer- 
tain mechanical assembly techniques as essential to successful job 
perfornuince probably could not properly rely on a training program 
that utilized exclusively an al)stract, academic or theoretical pre{jen- 
tation of the subject. 

57 . : ' 



that the only way in which the coirelation between Test 
21 scores and recruit school perforinaniee GOtflth t>e rele- 
vant was in terms of recriiit '^inability." The Court 
of Appeals concluded/ however, that. Test 2L did not 
predict ^^trainability"; it stated: , 

''[Blecause of the departnatental pol^y that nobody 
" fail Recruit School, appellees have i)ot shown that^the 
admission of applicants who score below 40* on 
Test '21 into Recruit Sthool would iwce^^sitate 'ex- 
landed: training time cn\produce*Remiit School ^fail- 
ures' 5J2 F.2d at 963 (emphasis added). 

After holding on narrow^ grounds that the Futransky 
Study was inadeqnale, however, the Court stated further 

that the Department had not: * . ** 

'> " ^ 

"persuaded us that trainability could be a projper 
^ - -^criterion foy validating Test 21. . . . fWje entertain 
grave .doubts wheth.er any of this type of evidence 
could be strengthened to the point of satisfying the 
heavy burden ' imposed "Tby Griggs,''' 512 'F.2d at 

feven more- fundamentally, the Court of Appeals sug- 
geste(J that validation against a training program may 
impermissible per sc. According to the Court, the ^*ultr- 
mate issue in this controversy" is whether proof '*'th&ft 
Test ^21 is .predictive of further progress" in the recruit- 
training program "is an acceptable substftute for a dem- 
onstration of a direct relationship betw^eeh performance 
on Test 21 and performance on the job.'" '512 F.2d at- 
rzMSs68. These portions of the Court of Appeals' .opinion 
deserve the careful attention of this Court. 

Eirst, the Court of Appeals- maj^have failed to jcon- 
sider carefully the -^job-relatedness" of "trainability" or 
success in training proghw^s under the applicable prece- 
( dents. The EEOC Guidelines>i:ovide that: 

"fWlork behaviors or other criteria of employee ade- 
quacy which the test is .intended to predict or iden- 



tify . . . may include measures , other than actual 
. work proficiency, such as training time. . . 29 
C.F.R. S 1607.01 b) (3). 

Further, in formulatii^ the job-relatedness standard in 
Grirjf/.s. this Court observed that the flinciamental flaw in 
the employer's proof was that neither test used in that 
case **\vas directe^d or intended to rneasure the ability to 
learn to perform a particular job or category of jobs/' 
401 U.S. at 428 f emphasis added j; In Dacis the De- 
partment has attempted to meet the vei7 test set out in 
(^rrif/f/s^ by demonstrating that Test 2^ was directed to or 
intended to measure ^^the ability to leaim" certain subject 
mattei- necessary to perform the job of a policeman. J/he 
Court's finding "that the Dep'iartment had not prov^eVthat 
low scorerjs on Test 21 expanded training time or in- 
creased Recruit School failures may be consistent ^ with 
the Guidelines and Griggs. The facts in the record 
cl<p not compel that conclusion, howevei', and the compet- 
inJ^'vU3fercnces were never considered by either the Dis- 
trir>t Count.or the Court of Appeals.'- In any event, the"^ 
Court 6f Appe'ate -^hotild have defo^ed iis holding more 



'-The I^jstyict Court did n^t address the question whether low 
scoro.s on 'tk^ 21 correlated w^th longer training time or increased 
tlie likehhoo(h>C^ecruit School failure. And, there .was no direct 
e\i(li'nce on tliK scCrtHi^ Ifo\vc\ er. recruits were required to achieve 
a passing sa)ro on each^'t)lKJje ei^ht Recruit School examniation.s 
aiui. aPr^uRh Recruit SchooM^ilurus were avoided, some candi- 
dates did not achieve passing s the first time aroun^k The 
Court of Appeals itself recogniztd^^M^t "if a paalicular (/^didate 
has ditncuity" ju-doing so, "he is er^at;^ta«^^^^L^ h/ succeeds 
in piis^HiK th(:.( xaminations/' 512 F.2d at l>d3. Given the corvelt^* 
tiun between low ^cores on Te.st 21 afid/low^iKd^'e]/^^^ in Re- 
cruit .School. It \yould be re<feoimble to infer that low T^st 21 ^corers 
were more likely to h*^ those for \ChorVi additional as.sj^^^ance was 
nec«ssar> The faet of no failuri'S was also used .^tt^drnnmistriUbe-;^ 
a lack of \alKlity of Test 21. iro\ve\(»r. since Test^--trnir?oTT^^ ' 
with .scores achieved biffin'' any assistance was provided, the fact 
of no failures is irrelevant to this issue. 
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-p^cisely so as not to appear to preclude, under any 
circumstances, reliance on training program success as a 
basis for concluding that a test is job-related. 

Second, the Court W Appeals never considered whether 
the* Department's training program was reasonably de- 
signed' to achieve proper employmei;it objectives. See 512^ 
'*F.2d at 965. To be sure, the reoord Jacks the detailed evi- 
dence regarding the program on basis of which such 
an evaluation should be made. In pa>fciciilar, evidence is 
lacking regarding those aspects of the ti^aJm^ program 
singled out by the Court of Appeals as diminishing its^* 
value as a basis for validating the use of Test ,2i^>^, 
the fact that no recruits are droif\)ed from the progi*ani 
for failure to pass one of the eight examinations. The 
Department did not offer any proof as to the reasons for 
this policy, although sufficiently rational reasons are 
immediately apparent and might have been the subject 
of an evidentiary showing/^ 

In the absence of such proof, this Court must 'decide 
whether the case is ripe for final disposition. Before 
validation of a test against a training program is either 
permitted' or ruled out automatically, this Courf 'may 
conclude that an employer should be required to relate . 
the training 'program to the employment attribute the 
program is professed to impart and to dernonstrate that 
his ^operation of the training program is reasOpble, fair, 



The cost of extra assistance for a recruit who has nearly com- 
^^pU'ted tlu' traininj? prog: ram may be substantially less than the cost 
(jf n'l/lacmjs' that^J'egruit with <i new applicant who AvoultLi^in the 
'tnuninf? program ane\v. thus the iio-failure policy may*be ocl^homic- 
ally efiicient. art her, the morale of the rccf*qit class or the >rencnil 
atmosphere in which the training program is conducted mi^ht be 
advi*-t:sely afTected by the possibility of some recruits being dropped 
from the progr^im caiusinp: ^the training program tu lose efficiency 
with respect to the group aaa^whole. a result the* Department ma> 
legitimately wish to avoid. ^ ) 

' GO 
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and suited to achieve that objective. If the Department 
cannot make such a showing in 'the Daci^ cas?, then its 
trailing program should -not suffice as a basis for deter- 
mining the "job-relat^dnes^'' of Test 21. 

Third, the breadth of- the Couit of Appeals' statements 
regarding ^^training programs'' is independently trouble- 
some. As emphasized earlier, the validity of tests under 
the^ applicable legal and professional standards neces- 
sarily, depends Qn the particular test in question and the 
circumstances, of its use. Standardized tests used for 
purposes otl^er than^ employment are sometimes validated 
again^t what might fall within an overly broad and ill- 
considered definition of a ''training program." For ex- 
ample, academic entrance examinations used in assessing 
sc•hala^tic aptitude are most frequently validated through 
ci^iterion-related studies, in which success in the curricu- 
lum for which the test is used in selecting, is widely 
regarded as the most useful criterion. In another con- 
text, professional school curricula are the touchstone jfor 
\alidating te^tb in the certification and licensing of pro* 
fessionals such as teachers and lawyers. The distinctions 
'between short pre-empdoyment training programs carried 
on by^individual employers and multi-year a'cademic cur- 
ricula seem obvious. Nevertheless, precision in analysis 
and language with respect to '^training programs'^ seems 
particularly important in this Court's review of the lower 
Court's statements relating to ''training programs'' and 
^trainability." . , 

r- 

:i Verbal aptitude as an appropriate key attribute 

of recruit training, *^ 
■* 

If the Recruit School program is adequately related 
to i\n imi)orJ,ant element of job performance, then the 
analysis mu^t consi/lei^~-further whether verba-l ability (re- 
gardless of how that attiitoe is measured j is itself 




suffia^tl^related to tRe^ training^^progi-am.^* If that 
link is satisf^torily established, then ^erbal ability ^ is 
necessarily related-tQ_job perfomance thi^gL the ti-ain- 
>rogram. 

Thr"affi3aVTts^^il^^. Mai^ L. TenopJ^nd Diane E. 
Wilson offered bytheTpefeiticg^^ contain cbnglusoi-y ob- 
servations in this^ regard. App^'Ir^M,.^ 

^ts offered by the respondgiits do riot>^arly dilute 
this^onclusion. App. ^9-57/ ^ 

The District Court considered the relationship between 
verbal ability (independent of how measured }^ and. the 
content of the training program as a link to job perfom- 
ance: 

' ''Study of the syllabus of the training course readily, 
demonstrates the intricacy of police proced}ires, the 
emphasis on report writing.^the. need^o differen- 
\tiate elements of numerous offenses and legal rul- 
ings, and the subleties of training required in be- 
havioral sciences .and related disciplines. Daily the 
significance of these skills demanding reasoning^ailiL- 
^ - > verbal and literacy skills is borne 'out in the^^ 
Cibld of the .criminal trial court. ^*-^-MajB\Supp7lCt>- 
17. ^ ^ ^ ' — - 

The Court of Appeals did not find it necessary to con- 
sider this link in the^ evidentiai-y chain because of -its 
view that verbal ability must be related directly to over- 



Two general avenues of protrf ^- o ^n^Hihr j ^ 
might offer proof as to the nature of the training program (dealing 
primarily with written materials, lectures, and report writing tech- 
niques) and expei;t testimony that this type of training program re- 
quires a certain Iev4 of verbal aptitude. The Department might also 
ch6ose a different route and prOvo that (1) Test 21 actually meas- 
ure3\verbal ability; (2) the eight Recruit School exan^i nations actu- 
ally measure the ^subject matter taught; and there exists a 
statistically significant correlation betw^dn scores on Te^t 21 and 
scores on the Recruit School examinations. This chain ftf^proof 
would permi't an inference that 'verbal ability wSs a key denieiit 
of recruit training. < ^ ^Ac^ 
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^11 job perfoi-mance. The question for this Court is 
whether there is a material issue of fact as to this 
point requiring the case to be remanded for the develop- 
ment of a more complete record at trial. 

B. Are the Tests Used by the Department Proper and 
Valid Measures of the Knowledge Desired by the 
Department? 

Furthei: analysis of the ' issues raised by the Davis 
case is required if the- Court concludes that (1) the* 
Department has properly identified knowledge of certain 
sifbjects as a central attribute of police work, and (2) the 
Department's training pix)grani is a reasoiiable mech- 
anism to impart that knowledge. At this- point, the **job- 
relatedness" determination in this case becomes largely a 
matter of technical test validation considerations. Four 
specific issues are raised by the tests used in the Davis 
case. ' . ^ 

1. Content validity of the Recruit School examinations. 

Under our apalysis of the issues, the Department can- 
not prevail unless the 6ight fexaminations used to meas- 
ure^"^proficiency in the Recruit School are valid. In the 
-absenee--^ such a fehowing, any correlation between 
scores on Test 21 and scores on the eight examinations 
cannot be relied upon in making the required "job-re- 
latedness" determination since the necessary link be- 
tween the knowledge being taught in the program and 
' the measurement tool does not exist. The required validity 
would be established through the technique of content 
validity, involving an iarrfalysis of the questions contained 
on the eight examinations and the curriculum actually 
i^ghtaiL_theULrain.in g program. Neit her court below 
addressed the need for this v3\idation; and the record, 
which contains the curriculum syllabus but not the eight 
examinations, may not permit its resolution by this Court. 
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The District Court apparently assumed that the ex- 
aminations given at the end of Recruit School validly 
assessed mastery of the topics taught. The District Court 
did not appear to consider that such a showing was re- 
quired (and could not be assumed) or that technical 
learning on test validation might be of assistance in 
making the determination/' The Court of "Appeals did 
not focus, as it might have, on the lack of content validity 
^to support its conclusion that the correlation between 
scores on Test 21 and scores on Recruit School examina- 
tions was an insufficient basis for finding the necessary 
"job-relatedness," 

2. Content validity of TM2L 



An apparently simple, but jjecessary, showing under 
oux analysis is that Test 21 actuaTly^oesopeasure ver- 
bal ability. This would be achieved in the same manner 
described above for the determination whether the Re- 
cruit School examinations actually measure the subject 
matter taught there. Both the District Court and the 
Court of Appeals seemed willing to assume that Test 21 
did measure vej'bal ability, perhaps from a perusal of 
the versions of the test contained in the record^ 

3, Correlation of Test 21^ with Recruit S^h<iol 
examinations* 

A separate issue raised by Davis is whether Test 21 
shows a sufficient correlation with the results on the 



ERIC 



''''Compare Pennsylvania v. O'NcilU 848 F. Supp. 1084 (E.D. Pa. 
1972), nff'd h]f an equnlly divided court, 473 F.2d 1029 CUl Cir. 
107*5 irnbanc)). whoro the court rojocted evidence of the job- 
relateclneys,uf a test Validated aprainst a police department training 
program on a variety, of grounds, one of which uas that^ 

"There is no e\*idence as to the type of examination adminis- 
tered at the Academy, If in fact tests of the type contained on 
the entrance exijmi nation were used, a high correlation could 
he anticipated (bven if an individual had not mastered the 
training material/' 848 F. Supp, at 109L 
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eight examinations. Both the parties and the courts be- 
low seem to agree that Test 21 has been validated as 'a 
predictor through the technique of criterion-related valid- 
ity — with the criterion yDeing scores on the eight Re- 
cruit School examinations — and that the correlation was 
sufficient and statistically significant under appropriate, 
professional standards. 

The Court of Appeals did. however, inappropriately 
suggest that no correlation betv^een two such examina- 
tions could ever suffice. Jhe Court stated that the De- 
partment's validation dataQ 

!'terids to prove nothihg more than that a written 
aptitude test will accurately pre^dict perfonnance on 
a second round of written examinations, and nothing 
to counter this hypothesis has been presented to us/' 
512 F. 2d at 962 (footnote omitted) ; 

In contrast, the APA Standards recognize that "fflor 
many employment . . . purposes" the **ideal criterion may 
be an achievement test. . . APA Standards at 34. As 
a legal matter, the decisions thus far have suggested that 
test-to-test correlation is appropriately questioned only 
where the tests are of a similar type. E.g., Pmrnylvania v. 
O'Neill 348 F. Supp. 1084 (E.D.Pa. 1972). The record 
in this case contains no suggestion that Test 21, an ap- 
titude test, was essentially similar to, or even remotely 
resembled, the eight achievement exams used to assess 
curriculurri masteiy.'" 



''\{t is equally unclear, from a leRal perspective, who boars the 
bunk^n of showing that "tost taking ability" accounted for a pi^r- 
ticular correlation. Nevertheless, in suggesting that ''nothing to^ 
counter this hypothesis" that the correlations reflected test-taking 
abilities "has been presented,^' the Court of Appeals purported to 
dispose of this cjuestion, too. At least one decision has suggested 
the contrai-y. Harper v. Mayor and City Council of Baltimore 350 F 
Snpp. 1187. 120r^ ilyM±)^.viodifird and aff'd sub iwm. Ilai-per v 
Klpsters, 48G F.2d IIM (4th Cir. 1973). 
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' 4. Validity bf the cut-off score on Test 21. 

The fourth measurement issue raised by the Davis 
case is one not addressed by either court below. The 
record reveals that Test 21 was used by the Department 
with a cut-off score of 40; long use seems to have been 
ithe principal explanation for this particular i?ut-off score. 
Ai)p. 191. Respondents appear not to have challenged the 
use of a cut-off score but to have proposed a lower cut- 
olff score of 35. 348 F. Supp. at 17. 

Under professional standards, most cut-off scores when 
used for employment selection must be validated, usually 
throuG:h a process independent of the validation of the 
test itself. The cut-off score must be demonstrated to 
classify persons with a reasonable level of probability as 
to those likely to possess the desired level of- verbal 
ability and those not likely to be so equipped/'' 

The record in Dam is silent on this issue and this 
Court may, therefore, wish to decide whether^ the De- 
partment's failure to justify the cut-off score provides 
^^rounds for affirming the Court of Appeals or for re- 
manding for the taking of evidence on this and the other 
issues in the case whicH would profit from further pro- 
cee(;lings below. 



Thore would he a similar issue wrth rospt'ct to the appropriate- 
ness of the c'lit-ofr score of 75 on the Recruit School examinations 
if that cut-ofT bcorc were used for classification J'uther than advisory 
purposes. 



CONCLUSION 



For the reasons stated above, this Court should set 
out a single analytical framework to be used in deter- 
mining the jai)-relatediiet;s of employment selection tech- 
niques. That framework shoukl be equally applicable 
to test and non-test methods of measuring employment 
selection criteria, should be designed to achieve basic 
objectives of rationality and fairness, and should be suf- 
ficiently flexible to permit a wide range of relevant evi- 
dence and to encourage co!(|^ued experimentation and 
growth in measurement technology. 

Respectfully submitted, 
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